QUICK START GUIDE

Follow these steps to register, launch, manage, and download the final gradebook for a Peer Assess Pro peer assessment using the Xorro Survey Management system.

Step / Xorro^[1]	Task / Reference Guide^[2]	Guide^[3]
REGISTER	once for your Xorro Teacher’s account ^[4]	1.1
LOGIN	to your Xorro Teacher’s account dashboard, at https://qf.xorro.com/	1.2
ORIENT	yourself to launching and managing a peer assessment activity.	1.3
PARTICIPANTS	arranged into teams from your class list Tip - adapt the sample Participants CSV file, here.	2.1
LAUNCH	a peer assessment activity by importing your Participants CSV.	2.3
MANAGE	your launched peer assessment activity	3.0
WARNINGS	alert you to take action with at risk teams and individuals.	3.1
TEAM RESULTS	entered as the basis for calculating a personal grade (optional).	3.3
SELECT	the Personal Result Calculation Method (optional)	3.4
REVIEW	class, team, individual feedback and grades.	3.5
PUBLISH	provisional and final feedback to team members.	3.6
FINALISE	the activity to prevent further survey responses.	4.0
DOWNLOAD	Finalised Teacher’s Gradebook, Qualitative and Teacher’s Feedback.	4.3

Download interactive at www.peerassesspro.com/quickstart-guide-for-teachers

Launching Peer Assess Pro™ using Xorro-Q: Overview

Once logged in to Xorro-Q, you launch a peer assessment activity.

During the launch process, you Import a Participants CSV that specifies team members (first, last name) arranged by their team name, login id and their email.

The Participants CSV is a comma-separated variable (csv) file that must contain the column headers shown in the example below.

Peer Assess Pro emails an activity URL that enables each team member to complete the peer assessments of their team members.

Timely reminders and personalised feedback reports are communicated to the students from Peer Assess Pro using the email addresses you provided in the Participants CSV.

Active Warnings on the Teachers Dashboard provide the teacher with advice about at-risk teams and individuals, poor peer assessment rating behaviour, and other progress indicators.

>>> View the comprehensive online and eBook ‘Get Started with Peer Assess Pro’

Example Peer Assessment Participants CSV File

first

last

team

group_code

AMTO01	Amanda	Tolley	Amanda.Tolley@noreply.com	Bear	BUS123.101/PMell/TutB/2020-05-28/SUM
ANWO08	Anna	Worth	Anna.Worth@noreply.com	Bear	BUS123.101/PMell/TutB/2020-05-28/SUM
HOBR03	Holly	Brown	Holly.Brown@noreply.com	Bear	BUS123.101/PMell/TutB/2020-05-28/SUM
ALJO11	Alice	Jones	Alice.Jones@noreply.com	Panda	BUS123.101/PMell/TutB/2020-05-28/SUM
GRGR15	Greta	Green	Greta.Green@noreply.com	Panda	BUS123.101/PMell/TutB/2020-05-28/SUM
JEWA06	Jeff	Wang	Jeff.Wang@noreply.com	Panda	BUS123.101/PMell/TutB/2020-05-28/SUM
BOWI12	Bob	Wilson	Bob.Wilson@noreply.com	Tiger	BUS123.101/PMell/TutB/2020-05-28/SUM
HEJO19	Henry	Jones	Henry.Jones@noreply.com	Tiger	BUS123.101/PMell/TutB/2020-05-28/SUM
JOSM13	John	Smith	John.Smith@noreply.com	Tiger	BUS123.101/PMell/TutB/2020-05-28/SUM

>>> Download example CSV, EXCEL, or Google Sheet

Example Survey Questions for a Team Member

Most Frequently Asked Questions (FAQs)

FAQ - Show me a quick video overview demonstration of the whole Peer Assess Pro system

FAQ - I’m having problems importing my participants csv

FAQ - How do I correct the Team Composition in a running peer assessment activity?

FAQ - What is the purpose of peer assessment?

FAQ - What questions are asked in the peer assessment survey?

FAQ - View the comprehensive ‘Get Started with Peer Assess Pro’ online eBook

>>> More FAQs at www.peerassesspro.com/frequently-asked-questions-2

Quick Link Map

Support, Feedback and Contact

Ask us for help, give us feedback, and request additional features.

https://www.peerassesspro.com/contact-us/

Patrick Dodd patrick@peerassesspro.com +64 21 183 6315

Peter Mellalieu peter@peerassesspro.com +64 21 42 0118 Skype myndsurfer

The Seven-Step Formula for Effective Peer Assessment

Download interactive poster https://www.peerassesspro.com/infographic/

Complimentary eBook

Mellalieu, P. J. (2020). How to teach using group assignments: The 7 step formula for fair and effective team assessment. Peer Assess Pro. https://www.peerassesspro.com/ebook

Teachers Process Flowchart: Overview

>>> Hyperlinked chart at http://tinyurl.com/papChart

PEER ASSESS PRO REFERENCE GUIDE

QUICK START GUIDE 1

Example Peer Assessment Participants CSV File 2

Example Survey Questions for a Team Member 3

Most Frequently Asked Questions (FAQs) 3

Quick Link Map 4

Support, Feedback and Contact 4

The Seven-Step Formula for Effective Peer Assessment 5

Complimentary eBook 5

Teachers Process Flowchart: Overview 6

PEER ASSESS PRO REFERENCE GUIDE 7

1. Login to your Xorro HOME page 24

1.1 First time users: Register 25

Getting started with Xorro Q 25

1.2 Login from your registered Xorro Account 26

1.3 Orient yourself to the Xorro HOME Dashboard 27

1.4 Orient yourself to the Peer Assess Pro platform 29

Fast video flyby 29

Review the Peer Assess Pro facilitators dashboard 30

Overview of the steps required to launch a peer assessment 31

1.5 Peer Assess Pro system flowchart detail 32

2. Launch Peer Assessment activity 33

2.1 Quick start launch 34

Participants CSV templates 34

2.2 Create the peer assessment Participants CSV 35

Alternative Participants CSV templates 35

Instructions and column explanations for the peer assessment Participants CSV 36

Requirements for a peer assessment Participants CSV file 38

Create a CSV version of your Participants CSV file 38

Why won’t Xorro load my Participants CSV file? 39

Your spreadsheet editor will typically NOT create a CSV file, unless... 39

Good practice hint: Create distinctive group codes for every peer assessment activity you launch 39

Large, multi-cohort streams in a class 40

Here There Be Dragons! 40

2.3 Launch and create the peer assessment activity 41

Select ACTIVITIES from the top menu bar 41

Launch Peer Assessment 42

Good practice hint: Avoid using the Xorro default Due Date 42

The Due Date date is advisory only 43

Initiate Create Activity 43

Point of no return! 43

View the Peer Assess Pro Teacher’s dashboard 44

Invite team members to respond and other automated activities 45

2.4 Use a Teamset Group to launch a peer assessment 48

From the Xorro HOME page select the PARTICIPANTS page 48

Select ‘Import Participants’ 48

Browse to your Team Members Group CSV file 49

Load, check and confirm correct team membership, then Import 49

Check class and team membership 50

3. Manage the Peer Assessment Activity 52

3.1 Action responses to warnings 53

Adjusting team composition 54

Note: Changes to a Xorro Group have NO EFFECT on current Team Composition 55

3.2 Automated and manual notifications 55

3.3 Enter Team Results 55

3.4 Select the Personal Result Calculation Method 57

3.5 Review class, team, and individual statistics 58

Review Class Results 58

Good practice hint: How to identify at risk students 59

The Individual Personal Snapshot 59

Four possible views of the Individual Personal Snapshot 62

Team Statistics 64

Qualitative Feedback 65

Teacher’s Feedback 65

Advanced Statistics 66

3.6 Publish provisional Personal Results to team members 67

Unpublished status 67

Published status 68

Results hidden when insufficient responses 68

4. Finalise the peer assessment activity 70

4.1 Why Finalise? 71

4.2 Publish Finalised Results to students 71

4.3 Download Teacher’s Gradebook of Results 72

4.4 Finalise the Activity … irrevocably! 73

FREQUENTLY ASKED QUESTIONS 74

Quick Link Map 74

FAQs for teachers 75

The peer assessment survey 75

Launch peer assessment activity 76

Manage the peer assessment activity 76

Responding to Active Warnings 77

Definitions, calculations, and examples 78

Miscellaneous 78

FAQs for team members 80

The purpose of peer assessment 80

Undertaking the peer assessment 80

Using the results from peer assessment for better performance 81

How peer assessment affects personal results 81

FAQ: What is the purpose of peer assessment? 83

Defining peer assessment 83

Developmental feedback 83

Determination of course personal result 83

Criteria for peer assessment in Peer Assess Pro™ 84

Peer Assess Pro assesses competencies valued by employers 84

Video 85

FAQ: When and how is the peer assessment conducted? 87

Formative assessment: optional but valuable 87

No surprises! 88

FAQ: How do I provide useful feedback to my team members? 90

FAQ: I believe I have been unfairly treated by the results of the peer assessment. How do I address my concern? 92

Symptoms of an unfair assessment 92

Steps to address an unfair peer assessment 92

A note on appealing a peer assessment result 93

Prevention is better than cure 94

FAQ: How do I interpret the feedback results I've received from the peer assessment? 96

FAQ: I don’t understand what my teammates are trying to tell me. How do I ask for better feedback? 97

FAQ: What steps can I take to get a better personal result? 98

Raise your Team Result 98

Use your institution’s academic support services 98

Raise your Peer Assessed Score 99

How do I address proactively the challenges of team work? 99

Learning constructively from mid-course peer assessment feedback 100

FAQ: What happens if I try to 'game' (fool? play? disrupt?) the peer assessment process? 102

The tricks we know! 102

Examples: Highly specific and individualized information 103

1. Low quality assessor rating 103

2. Low quality team rating 103

3. Outlier individual rating 104

4. Mismatched self-assessment 104

5. At risk team member 105

Example: Better feedback. Better teams 105

Which teams will raise the Active Warning: Low quality team rating? 106

Which teams tend to have a higher team result? 107

Which teams have worked most productively as a team? 107

Active Warnings, thresholds parameters, and program logic 109

FAQ: Give me a quick overview of how to launch a Peer Assess Pro™ activity through Xorro 110

FAQ: What are the design objectives, key features, and benefits of the Peer Assess Pro development? 112

Design objectives 112

Benefits for students 112

Benefits for teachers 112

Peer Assess Pro™ is a work in progress 113

Where’s the latest? 113

FAQ: How do I find the Peer Assess Pro Xorro Teacher’s dashboard? 115

HOME: Running Activities 115

Alternative method: ACTIVITIES: Running Activities 115

FAQ: How do I navigate the PARTICIPANTS page for Peer Assess Pro? 118

Select PARTICIPANTS Tab 118

Orientation note: Select an existing Group 119

Inactive functions in PARTICIPANTS page 120

FAQ: How do I correct the team composition in a running peer assessment activity? 121

Adjust the team composition in a running peer assessment activity on an LMS 122

Take care! Here there be dragons!! 122

Overview 122

What happens with Synchronise All? 123

Precautions before Synchronise All! 123

Some survey responses might be deleted! 124

Team composition view prior to synchronise all 125

Team composition following synchronise all 126

Survey notifications 126

Adding orphan teams to a running activity 127

Why was my Synchronise All action rejected? 127

Good practice hint when creating a peer assessment activity 127

LMS team arrangement facilities for Moodle 128

Adjust the team composition in a running peer assessment activity on Xorro 129

Take care! Here there be dragons!! 129

Key check points 129

View the team composition 129

Correct the team composition 130

Subtle technical note 131

FAQ: Can I create a peer assessment activity without having all my teams correctly identified by team name and/or team membership? 132

FAQ: How do I create a CSV file from a Google Sheet? 133

Good practice hint! 133

Sample of participants csv file opened using a text editor 134

FAQ: How do I view a demonstration version of Peer Assess Pro? 135

FAQ: How do I correct the participants (team members) in a group already uploaded to Xorro? 136

Update a group’s team members for future use 136

Correct the team members associated with an existing Xorro TeamSet Group 137

FAQ: Where may I view the most recent version of the user guides? 139

Quickstart Guide 139

Video guides 139

Latest reference guide 139

Work in progress Google DOCS development version 139

Frequently Asked Questions for teachers and team members 140

Teachers Process Flowchart 140

FAQ: How do I decide which Personal Result method to apply in my peer assessment activity 141

FAQ: How are peer assessment and personal results calculated and defined mathematically? 147

Calculation methods that exclude a team result 148

Calculation methods that incorporate a team result from team outputs 148

FAQ: How do students know where and when to complete the peer assessment activity then review their results? 149

Automated communications to students 149

Standard operating mode 149

Alternative mode for student access to assessment and results 150

FAQ: How do I view and experience what the students experience? 151

View your student’s personal results feedback report directly from your Teacher’s Dashboard 151

View your students’ experience of the Peer Assess Pro™ survey 151

Enter your Participants’ URL into your browser 152

Select the activity you wish to experience 153

View a survey ready and waiting for responses 154

View a sample question 154

View a student’s published results 155

View the peer assessment survey for a demonstration class 157

FAQ: Why are different terms used to display peer assessment results in the Xorro and previous Google versions of Peer Assess Pro™? 158

FAQ: How do I take action on the Warnings presented in the Peer Assess Pro™ Teacher’s Dashboard?’ What if I ignore the Warnings? 159

Critical and catastrophic warnings! 159

Important warnings 159

Informational warnings 160

Optional emails generated for team members 160

FAQ: When, why, and how do I Refresh and Update Results? 161

When to recalculate 161

Why recalculate? 161

How to recalculate 162

FAQ: What questions are asked in the peer assessment survey? 163

Example Peer Assessment Survey: Quantitative 164

Example Peer Assessment Survey: Qualitative 165

FAQ: How is the Peer Assessed (PA) Score calculated? 166

The self-assessment is excluded from calculating PA Score 166

Mathematical definition of Peer Assessed Score, PA Score 167

Example calculations of Peer Assessed Score 170

Alternative mathematical formulations of PA Score 172

Calculation from Average Rating 172

Calculation from Average Team and Leadership Contributions 173

FAQ: How is the self-assessment used to calculate Peer Assessed Score? 174

Spider chart of individual and averaged team peer ratings 174

Index of Realistic Self-Assessment (IRSA) 175

FAQ: How is the Peer Assessed Index (PA Index) calculated? 176

Mathematical definition of Peer Assessed Index 176

Example calculations of Peer Assessed Index 177

FAQ: How is the Indexed Personal Result (IPR) calculated? 179

Mathematical definition of Indexed Personal Result 179

Example calculations of Indexed Personal Result 180

FAQ: How is the Normalised Personal Result (NPR) calculated? 182

Mathematical definition of Normalised Personal Result 183

Example calculations of Normalised Personal Result 184

Impact of adjusting the Spread Factor on Normalised Personal result 186

FAQ: How is the Rank Based Personal Result (RPR) calculated? 188

Mathematical definition of Rank-Based Personal Result 189

Example calculations of Rank-Based Personal Result 191

Example calculation with tied ranks 193

Alternative mathematical formulation 193

Adjusting the range using a spread factor 194

Example calculation with spread factor 195

FAQ: How is the Rank Based Personal Result (RPR) calculated (Pre-2022)? 196

Mathematical definition of Rank-Based Personal Result 197

Example calculations of Rank-Based Personal Result 198

Example calculation with tied ranks 200

FAQ: How is Standard Peer Assessed Score (SPAS) calculated? 201

Design features of Standard Peer Assessed Score 202

Mathematical calculation 203

Example calculations of Standard Peer Assessed Score 205

Example charts for Standard Peer Assessed Score 208

Assumptions about Standard Peer Assessed Score 209

The impact of gaming peer assessment 210

FAQ: What is the influence on Standard Peer Assessed Score (SPAS) if a team rates ALL its members with a Peer Assessed Score of 100? 211

FAQ: Would a student receive the same Standard Peer Assessed Score (SPAS) if rated in another class? 211

FAQ: What is Employability? How is it calculated? 213

Mathematical calculation of Employability 213

Conditioning transformations to de-emphasise unsubstantiated precision 214

Example calculations of Employability 214

FAQ: How is the Index of Realistic Self Assessment (IRSA) calculated? 217

Mathematical definition of the Index of Realistic Self Assessment 217

Example calculations of the Index of Realistic Self Assessment 218

Why an IRSA of 100 is not a perfect score! 219

FAQ: How do I interpret measures of realistic self-assessment? 221

Interpreting the Index of Realistic Self Assessment (IRSA) 221

Typical IRSA 221

Overconfident IRSA 221

Underconfident IRSA 221

Developing an exceptionally realistic self-image, ERSA 222

What are the benefits of having an Exceptionally Realistic Self Assessment? 222

What can get in the way of having an Exceptionally Realistic Self-Image? 222

How do I develop my Exceptionally Realistic Self-Image, ERSI? 223

FAQ: How is insignificant team agreement identified? WARNING 0041 225

Warning detail 225

Recommended action 226

Award the same personal result 226

Special cases 227

The same personal result awarded updates according to teacher’s adjustments to team result or result method 227

‘Award the same personal result’ status may be deactivated automatically 228

‘Exclude from calculations’ overrides actions that address insignificant team agreement 229

Defining concordance 229

Interpreting the concordance value, W 230

Exclusion of self-assessment ratings 230

Tied rankings 230

Significance of the concordance statistic 231

Requirement for calculating concordance 231

Team discrimination table shows class concordance statistics 232

Example calculations 233

Example A. High rating agreement amongst the team members 233

Ex A. Calculation of Concordance, W, from ranks 235

Conclusion for Example A 235

Example B. Low agreement amongst the team members 236

Conclusion for Example B 237

Example C. Teammates show low discernment by submitting tied ranks 238

A note on computational efficiency 240

FAQ: How is an outlier peer assessment rating identified? WARNING 0042 243

Warning detail 243

Failure to agree across the whole team 243

Example calculations 244

Threshold for warning of outlier individual peer rating 245

Alternative mathematical calculation of Assessor Impact 246

Alternative example calculations 246

FAQ: What is a mismatched self-assessment (IRSA)? WARNING 0040 248

Warning detail 248

Threshold for warning of mismatched self-assessment 249

Example calculations 249

Recommended action for facilitator 250

FAQ: What is an adjusted team arrangement request? WARNING 0006 251

Warning detail 251

Available actions 251

Special cases 252

Reassign a new participant to an existing or new team 254

FAQ: What is an inactive team member? WARNING 0048 256

Warning detail 256

Available actions 256

Good practice for addressing an alleged inactive student 257

Special cases 258

Reassigning a student to a new ‘team of one’ 258

Exclude from calculations overrides actions that address insignificant team agreement 258

Example of exclusion from calculations 258

Impact of exclusion from calculations 262

FAQ: What is a low quality team rating? WARNING 0050 263

Warning detail 263

Threshold for warning of low quality team rating 264

Example calculations 264

Recommended action for facilitator 265

High performing teams 265

Example case 265

FAQ: What is a low quality assessor rating? WARNING 0300 267

Warning detail 267

Threshold for warning of low quality assessor rating 267

Example calculations 268

Recommended action for facilitator 269

High performing teams 269

FAQ: What is a valid assessed team? WARNING 0022 270

Warning detail 270

Results not displayed to members of non-valid assessed teams 270

How many valid and invalid teams do I have? 271

Recommended action for facilitator 271

Mathematical definition 271

Example calculations 272

FAQ: What is an ‘at-risk’ team member? WARNING 0036 273

Warning detail 273

Recommended action for facilitator 274

Threshold for warning of ‘at-risk’ team member 274

Example calculation 275

Example calculation 278

Limitation 279

Alternative approaches to identifying at risk students 282

FAQ: What is a team with low psychological safety? WARNING 0034 286

Warning detail 286

Definitions 286

Example calculation for Team Safety 287

Recommended action for facilitator 288

FAQ: What is an unsafe team member? WARNING 0035 291

Warning detail 291

Recommended action for facilitator 291

Limitations 292

FAQ - What emails have been sent by the platform? 294

Survey notifications history 294

Track-and-trace of emails to participants 294

FAQ: What is the content of emails sent by Peer Assess Pro to Participants? 297

Preview email from Active Warnings 297

Preview all emails available for sending 297

Table of email subjects sent to participants 299

Table of email body text sent to participants, listed by Email ID and Subject 301

FAQ: What is the content of emails sent by Peer Assess Pro to Facilitators? 311

Table of email subjects sent to facilitators 312

Table of email body text sent to facilitators, listed by Email ID and Subject 313

FAQ: How do I login to my peer assessment Activity URL 321

Activity URL 321

Participant URL 321

Successful login through Activity URL 322

FAQ: I am unable to login. My login failed 324

Investigation and remedies for login failure 324

You entered your ID incorrectly. 324

Your teacher or facilitator has entered your ID incorrectly 325

The Xorro Activity related to the Activity URL has not yet reached its Start Date 325

The Xorro Activity related to the Activity URL has been Finalised and Finished. 326

The Xorro Activity related to the Activity URL has been Abandoned 327

The institution manager for Xorro has not maintained payment of the subscription to use Xorro and/or Peer Assess Pro 327

An exceptional system fault has occurred with the Xorro participants database entry for your ID: duplicate identical ids 328

FAQ: Can I adjust the Start Date or Due Date for a running activity? 330

Adjusting the Start Date 330

Adjusting the Due Date 330

The good news: The Due Date date is advisory only 331

Advise students of your extended deadline 331

Worst case scenario: Abandon the peer assessment 331

FAQ - I’m having problems importing my participants csv 334

What are the common problems when importing a participants file? 334

Is my Participants CSV in the correct format for launching a peer assessment activity? 336

Sample of participants csv file opened using a text editor 336

Sorted Participants CSV viewed in Google Sheets 337

Extra for experts 338

Key points! 338

Error notifications upon upload of a participants CSV to a peer assessment 339

Examples of error notifications upon upload of a participants CSV to a peer assessment 340

Good practice hint! 340

Comprehensive list of potential errors when attempting to import participants csv 341

Potential errors in a participants csv 341

About Comma-separated values (CSV) 343

FAQ - Problems editing and creating participants CSV files 345

Trouble opening csv files with Excel due to regional settings 345

Explanation 345

Workaround solution 346

Solution 1: Use a simple text editor to find and replace semicolons with commas 346

Illustration using Mac TextEdit 346

Solution 2: Adjust Language and Region to use point as decimal marker 350

Illustration using Mac OS on an Apple computer 350

FAQ - How do I fix an invalid, missing or failed email delivery? WARNING 0026 354

Corrective action: avoid these emails 354

Corrective action: use these emails 355

FAQ - How do I resolve an unsynchronised team arrangement? ACTIVE WARNING 0021 357

Corrective Action 358

What is a team arrangement? 358

Synchronise all team arrangements 359

Here there be dragons! 360

Postpone 360

An example scenario 361

One week later 362

Alert to unsynchronised team arrangement 362

Unsynchronised Team Composition 363

Teacher’s actions 364

Before synchronisation 364

Successful synchronisation of the corrected team arrangement 365

FAQ - What is the benefit of a standardized peer assessment rubric? 368

FAQ - What if a student mistakenly advises they are in an incorrect team? 370

Team membership confirmation by student 370

Mistaken team membership notification 370

1. Login to your Xorro HOME page

1.1 First time users: Register

Register a new Xorro Teacher’s Account as a Free Facilitator

https://www.xorro.com/free_accounts/pap/new

Getting started with Xorro Q

For related information relevant to registering as a new facilitator:

Getting Started with Xorro Q

For further details contact Patrick Dodd at the offices of Peer Assess Pro.

1.2 Login from your registered Xorro Account

Browse to: http://qf.xorro.com
Enter the Username and Password with which you were registered.

After you login, The your Xorro HOME Dashboard page shows will display, as shown in Section 1.3 Orient yourself to the Xorro HOME Dashboard

Now proceed to follow the steps in the Quickstart Guide, or the detailed explanations in Section 2. Launch Peer Assessment Activity

Quick links and related information

VIDEO: Login and orientation

View: Quick Start Guide

Section 1.3 Orient yourself to the Xorro HOME Dashboard

FAQ: How do I find the the Peer Assess Pro Teacher’s dashboard?

1.3 Orient yourself to the Xorro HOME Dashboard

Your Xorro HOME Dashboard page shows

Along the top right:

HOME - this page!
ACTIVITIES - All Running and the results from previous Activities you have executed and are Finalised and Expired..
PARTICIPANTS - Here you view and import a class list of participants allocated into their teams, a Participants CSV.

Down the left:

RUNNING ACTIVITIES - shows peer assessment and other Xorro Activities you have Created and Launched, but are not yet Expired
RECENT PREPARED ACTIVITIES - activities you have created, but which have not yet Started. An unstarted activity means team members cannot view, login, access or complete the activity through their Participant’s URL. The activity will not yet be listed.
RECENT RESULTS from Expired activities. A peer assessment activity is Expired automatically two weeks after it is FINALISED by the teacher from the Peer Assess Pro dashboard. Results from an Expired Activity can no longer viewed by a student.

Participant URL - A team member can find ALL your current RUNNING ACTIVITIES at this URL. Follow the Participant URL to experience how a student sees Peer Assess Pro.
When in doubt, lost, or confused select the HOME page or ?HELP.

Quick links and related information

VIDEO: Login and orientation

View: Quick Start Guide

FAQ: How do I find the the Peer Assess Pro Teacher’s dashboard?

Peer Assess Pro system flowchart detail

Peer Assess Pro system flowchart detail http://tinyurl.com/papChart

Each process box in the flowchart pdf version of the flowchart links directly to the specific page in this Reference Guide that explains that step in the process.

1.4 Orient yourself to the Peer Assess Pro platform

Fast video flyby

View this short video illustrating many of the features, benefits, and processes involved in using the Peer Assess Pro platform (6 minutes).

http://tinyurl.com/digitalFlyBy

Review the Peer Assess Pro facilitators dashboard

These are the key features of the Facilitators Dashboard accessed through the Xorro ACTIVITIES tab when you have launched a peer assessment activity.

Middle tabs - Peer Assess Pro, Team Composition and Team Results adjustment
Available Actions - Where you change the Personal Result Method, Publish survey results to participants and Download gradebook statistics
Active Warnings - Where you manage the activity by responding to exceptional conditions, such as incomplete or potentially unconstructive survey submissions

Overview of the steps required to launch a peer assessment

1.5 Peer Assess Pro system flowchart detail

PDF with hyperlinks at Xorro Peer Assess ProTM Teachers Process Flowchart http://tinyurl.com/papChart

2. Launch Peer Assessment activity

2.1 Quick start launch

Create a file containing your class list that shows every team member organised into their teams. The required file format is Comma Separated Variables (CSV). This is your Participants CSV file. A sample of the file format is shown in Section 2.2 Create the peer assessment Participants CSV file

Use any of these following templates to adapt and create your Participants CSV file using your preferred editor.

After editing the template, remember to create a CSV file type using SAVE AS CSV, DOWNLOAD AS CSV or EXPORT AS CSV, depending on your spreadsheet editor.

Participants CSV templates

Excel sheet

Google Sheet

CSV file

For a registered Xorro user, use this link to launch a new peer assessment activity. You will be presented with an option to import directly your Participants CSV.

https://qf.xorro.com/pap/launches/new

If your CSV refuses to load, or the activity fails to create, review the detailed steps in the next sections to ensure your CSV is specified correctly.

FAQ - I’m having problems importing my participants csv

Check carefully that the specifications detailed in the INSTRUCTIONS and COLUMN EXPLANATIONS presented within the sample.csv template are followed strictly.

2.2 Create the peer assessment Participants CSV

Use a spreadsheet editor, such as Google Sheets, Excel or Numbers to produce a file that contains columns of data with these column headers id, first, last, email, team, and group_code. Precise INSTRUCTIONS and COLUMN EXPLANATIONS for each of these data are detailed below.

Alternative Participants CSV templates

Use any of these templates to adapt and create your Participants CSV (comma separated variables file) using your preferred editor. The templates contain the example data and instructions shown below.

CSV file

Excel sheet

Google Sheet

In the sample files, only the group BUS123.101/PMell/TutB/2020-05-28/SUM is a valid teamset suitable for processing by Peer Assess Pro. This is the only group that specifies membership of teams by the students in the class, the teams being Panda, Bear and Tiger.

Sample peer assessment Participants CSV

id	first	last	email	team	group_code
ANWO08	Anna	Worth			ARTS123.204/WShak/2021-02-28
GRGR15	Greta	Green			ARTS123.204/WShak/2021-02-28
AMTO01	Amanda	Tolley	Amanda.Tolley@noreply.com	Bear	BUS123.101/PMell/TutB/2020-05-28/SUM
ANWO08	Anna	Worth	Anna.Worth@noreply.com	Bear	BUS123.101/PMell/TutB/2020-05-28/SUM
HOBR03	Holly	Brown	Holly.Brown@noreply.com	Bear	BUS123.101/PMell/TutB/2020-05-28/SUM
ALJO11	Alice	Jones	Alice.Jones@noreply.com	Panda	BUS123.101/PMell/TutB/2020-05-28/SUM
GRGR15	Greta	Green	Greta.Green@noreply.com	Panda	BUS123.101/PMell/TutB/2020-05-28/SUM
JEWA06	Jeff	Wang	Jeff.Wang@noreply.com	Panda	BUS123.101/PMell/TutB/2020-05-28/SUM
BOWI12	Bob	Wilson	Bob.Wilson@noreply.com	Tiger	BUS123.101/PMell/TutB/2020-05-28/SUM
HEJO19	Henry	Jones	Henry.Jones@noreply.com	Tiger	BUS123.101/PMell/TutB/2020-05-28/SUM
JOSM13	John	Smith	John.Smith@noreply.com	Tiger	BUS123.101/PMell/TutB/2020-05-28/SUM
THWI18	Thomas	Windsor	Thomas.Windsor@noreply.com	Tiger	BUS123.101/PMell/TutB/2020-05-28/SUM
ANWO08	Anna	Worth	Anna.Worth@noreply.com		COMP123.201/PDod/TutA/2020-10-01
HOBR03	Holly	Brown	Holly.Brown@noreply.com		COMP123.201/PDod/TutA/2020-10-01
JOSM13	John	Smith	John.Smith@noreply.com		COMP123.201/PDod/TutA/2020-10-01

Instructions and column explanations for the peer assessment Participants CSV

INSTRUCTIONS

1. Organise your participants data into the columns corresponding to those shown in columns A to F, the first 6 columns headed 'id' through 'email'.

You might find it helpful to paste your data from row 17, below the sample data provided in rows 2 through 16.

The sample data provided demonstrates ten unique individuals (ids), organised into three different groups. One group contains a further three teams.

A group might comprise all members of a class, or subdivisons such as streams, cohorts, sections, or tutorial groups.

In the group called BUS123.101/PMell/TutB/2020-05-28/SUM the participants are subdivided further into three different teams, Bear, Panda and Tiger.

Only group BUS123.101/PMell/TutB/2020-05-28/SUM is a Xorro teamset suitable for a peer assessment activity.

A group is not a team. A group (such as a class) may contain several teams, in which case that's a Xorro teamset.

2. If you are preparing a separate file, ensure you use exactly the same column headers for your list as shown in row 1.

That is, 'id', 'first', 'last', 'email', 'team', 'group_code'.

These headers are not case sensitive.

The sequence of column headings is NOT IMPORTANT.

You may optionally include additional headers and columns of data. This data will be ignored by Xorro.

Data may be sorted by any of the columns.

3. Read carefully the COLUMN EXPLANATIONS, below, for each type of data.

Some data is optional, and can be skipped, as shown for group_code ARTS123.204/WShak/2021-02-28

4. Delete the sample data, immediately below the header row.

That is, delete everything between row 2 and row 16.

CRITICAL: CHECK you do not have duplicate ids in the same group_code.

CHECK you do have all the ids in your class allocated to to a group, and, optionally, a team

5. If you have used this page as your template, you may DELETE this 'instructions' column.

That is, delete anything not part of your data.

Keep the column headers. The headers must be on row 1 of your file.

6. Save (Download, Export, Save As) the file as a CSV, giving it an appropriate filename.

7. From Xorro-Q, browse to PARTICIPANTS, then upload the CSV file.

Alternatively, when you Launch a Peer Assessment Activity, you can IMPORT directly the CSV to create or update the activity.

From this sample file, upon upload three groups would be created in Xorro: ARTS..., BUS.... and COMP....

Only one of the groups is a teamset containing the three teams Bear, Panda, and Tiger.

8. COLUMN EXPLANATIONS

id - Compulsory field.

Identifier for this participant, must be unique for the entire institution.

For a peer assessment activity, this is the participant's login id.

No blanks or characters such as #@$%&*()+

first - Compulsory field.

Participant's first name

last - Compulsory field.

Participant's last name

email - Optional field.

The participant's email.

Ideally required for a peer assessment activity when you require autogenerated warnings and notifications from Peer Assess Pro.

team - Optional field. Required for peer assessment activity.

The name of the team in which the participant is a member.

The participant can be a member of NO MORE than one team within the same group.

A participant may belong to different teams in different groups.

group_code - Optional field. Required for a peer assessment activity.

The code for the group (ie course, class, stream, cohort) into which the participant is being enrolled.

If the participant is in multiple groups, supply a separate line for each group in which the participant is a member.

Good practice. Append to your root code, such as BUS123.101, abbreviations that indicate the teacher, activity date (start or due), subdivision (stream, cohort), summative or formative.

Note that Anna Worth is enrolled in three groups and in one team within group BUS123.101/PMell/TutB/2020-05-28/SUM

Requirements for a peer assessment Participants CSV file

The column headers must be represented exactly in any sequence. Namely, ‘id', 'first', 'last', 'email', 'team', 'group_code'.
All students in the same class will usually have the same Xorro group_code designation. Typically, use your class code extended as detailed in the Good Practice Hint below. For large classes, review the section below

Large, multi-cohort streams in a class.

Check for duplicate ids in your CSV, and REMOVE. A student with the same id cannot, must not, shall not be allocated to more than one team within the same grouping
Ensure people with similar names, such as John SMITH and Cheung WANG have different ids.
Ensure teams have 3 or more team members. Peer assessment results will not be produced for teams with 0, 1, or 2 members. These team members will receive the Team Result.
1123 ABC is an invalid id as it has a space between the characters. 123ABC is a valid id. #123ABC is an invalid id.
The email data can be used in the id column. Just copy it over.
The file type required for import into Xorro is CSV, which means comma separated variables. You can download and view an example CSV file as described in the section Alternative Participants CSV templates

Create a CSV version of your Participants CSV file

After editing the template, remember to create a CSV version of your file. Depending on your editor, the appropriate command is:

FILE… SAVE AS … TEXT CSV

FILE… DOWNLOAD AS … Commas separated values (.csv)

FILE… EXPORT AS CSV

FILE… EXPORT TO… CSV

Why won’t Xorro load my Participants CSV file?

First, see FAQ - I’m having problems importing my participants csv

Your spreadsheet editor will typically NOT create a CSV file, unless...

In frequent cases, using the FILE… SAVE command in your spreadsheet editor will produce a file with the incorrect file format, such as .xls, .sheet, or .numbers.

Xorro will reject those file formats. Xorro accepts and loads only .csv.

Follow this advice

FAQ: How do I create a CSV file from a Google Sheet

Good practice hint: Create distinctive group codes for every peer assessment activity you launch

We advise creating a new, unique group_code for each Xorro Activity you create, even for repeat peer assessments within the same class term or semester.

Use a group_code like this

BT123.101/PJM/2020-03-28/FORM

We suggest your group_code include these elements as per the example above:

The class code you use to designate the class or course, eg BT101
The teacher’s initials or name, eg PJM
The due date or start date for the activity. eg 2020-03-28
Optionally, annotate if the assessment is formative (FORM), mid course, or end of teaching, summative (SUM).

We recommend your resulting group_code should distinguish uniquely this semester’s mid-semester formative peer assessment(s) from last semester’s end of class summative where, perhaps, the same institutional class code could have a different set of student names.

The group_code is specified in the Participants CSV file you import prior to launching a Peer Assess Pro™ Activity.

Large, multi-cohort streams in a class

In the general case, a very large class could comprise several cohorts, streams or tutorial sets, each subclass containing several teams conducting one or more peer assessment activities. Consequently, your group_code should help distinguish these separate peer assessment activities. For example,

BT123.101/PJM/TutB/2020-05-28/SUM

Here There Be Dragons!

Consider two teachers at the same institution teaching the same course but with different tutorial groups. If they use the same goup_code, such as BT101, they will load their own team sets into the same Xorro Participants’ Group, additively, thereby causing mutual confusion and dismay. Similarly, a teacher using the same group_code from term to term, semester to semester, and year to year will experience similar grief.

Quick links and related information

FAQ - I’m having problems importing my participants csv

FAQ: How do I correct the Team Composition in a running peer assessment activity?

FAQ: How do I correct the participants (team members) in a group I uploaded?

FAQ: How do I create a CSV file from a Google Sheet

2.3 Launch and create the peer assessment activity

In summary

Select ACTIVITIES from the Xorro top menu bar
Launch Peer Assessment , in which you will Import the Participants CSV created earlier.
Initiate Create Activity
View the Peer Assess Pro Teacher’s dashboard
Peer Assess Pro will automatically invite students to respond once the Start At date has been reached.

Select ACTIVITIES from the top menu bar

Launch Peer Assessment

Enter the following details, in this sequence

Activity Title - A description that will be meaningful to your students, such as the class name and assessment type eg BT101 Mid-semester Peer Assessment - Applied Management
Participants CSV - Import - Identify and load the Participants CSV file that you created in Step 2.2 here.
Note the Group Code will be determined automatically from the Imported Participants CSV uploaded in the previous step.
Start At - the date:time from which the students will be able to view and complete the peer assessment. Once you have Created the Activity, students will NOT be able to view the activity or submit their responses UNTIL the Start date.
Due Date - this is the date that students will be advised by which they should complete the peer assessment.

Good practice hint: Avoid using the Xorro default Due Date

Set a realistic Due Date that is your target for when you expect and want most students to have completed the peer assessment. In practice, typical Due Dates are set to within four days to seven days beyond the Start Date.

The Due Date is used by Peer Assess Pro to generate automatically:

Email reminders to students to complete the assessment
Prompts to the teacher to conduct important administration and management activities during the activity, and prior to Finalisation

If you use the Xorro default Due Date, which currently is NOW, the Start Date, you will not receive the benefits of the automated processes conducted by Peer Assess Pro that are triggered by a practical Due Date.

The Due Date date is advisory only

The ‘Due Date’ date is advisory only. Students can CONTINUE to submit responses beyond the Due Date UNTIL the teacher Finalises the activity. After the Finalisation Date, the students will have no more than two weeks to review their results.

FAQ: How do I adjust the Due Date or deadline?

The short answer is ‘You can’t adjust the Due Date!’ You don’t need to!

Initiate Create Activity

After setting the Start At and Due Dates, select Create Activity .

Point of no return!

Double check your Start Date and Due Date carefully!

Once you Create Activity you cannot adjust the Start Date. The peer assessment Survey and the Email notifications to students requesting their response are created when the Start Date is reached. Furthermore, the email advises the students the Start Date and Due Date.

Therefore, an adjustment to the Start Date would confuse the students as the Participant Activity URLs would be announced to students. Those Activity URLs could become unavailable to the students if dates were adjustable.

For a similar reason, you cannot adjust the Due Date. However, the ‘Due Date’ date is advisory only. Students can CONTINUE to submit responses beyond the Due Date UNTIL the teacher Finalises the activity.

FAQ: Can I adjust the Start Date or Due Date for a running activity?

In short, No!

In a ‘worst case scenario’ you can abandon the activity and launch a new activity. Review the foregoing FAQ for details on how to Abandon a running Peer Assess Pro activity.

View the Peer Assess Pro Teacher’s dashboard

The Peer Assess Pro Teacher’s Dashboard will now show for this newly created activity
Note the Activity Title, and Days remaining until the Due Date for student’s completion
Note the Active Warnings. These advise you of actions you must take to deal with a variety of circumstances that may require remedies.
The created activity now appears as a RUNNING ACTIVITY even though the Start Date may not have been reached. It is RUNNING so far as the Teacher-Facilitator is concerned, enabling you to undertake several management activities explained in Section 3. Manage the Peer Assessment Activity

Peer Assess Pro Teacher’s Dashboard

Invite team members to respond and other automated activities

When the Start Date occurs, Peer Assess Pro automates several activities:

A peer assessment survey is composed, unique for every team and every student.
Only when the Start Date is reached the survey is available for students’ access through the Activity-Specific Participant URL
An email is sent to all Team Members requesting them to complete the peer assessment survey before the Due Date. The Activity-Specific Participant URL is included in the email.
The Activity’s status will be listed as a RUNNING ACTIVITY in the HOME and ACTIVITIES pages of Xorro.
The Xorro Team Survey app begins to collect students’ responses to the survey, conveying the responses to the Peer Assess Pro database, report analyser, and Peer Assess Pro dashboard within Xorro.

A unique peer assessment survey is created for every team and team member

Quick links and related information

FAQ - I’m having problems importing my participants csv

FAQ: How do I correct the Team Composition in a running peer assessment activity?

FAQ: Can I adjust the Start Date or Due Date for a running activity?

FAQ: How do I view a list of the participants (team members) in the group I uploaded?

FAQ: How do students know where and when to complete the peer assessment activity then review their results?

FAQ: How do I view and experience what the students experience?

FAQ: How do I find the the Peer Assess Pro Teacher’s dashboard?

2.4 Use a Teamset Group to launch a peer assessment

This is an alternative approach to launching a peer assessment activity. This is a two stage process where you can

Upload a Participants CSV creating one or more TeamSet Groups
Launch a Peer Assessment activity during which you
Select the previously-imported TeamSet Group, then
Create the Peer Assessment activity.

From the Xorro HOME page select the PARTICIPANTS page

(Image to come)

Select ‘Import Participants’

This uploads your Participants CSV within which you have classified your students into teams, as detailed in Section 2.2 Create the peer assessment Participants CSV file

Note that multiple teamset groups may be created using this import process. This is potentially useful for managing peer assessment in large, multi-stream classes.

Browse to your Team Members Group CSV file

Load, check and confirm correct team membership, then Import

You should see a list of all the students belonging to the class for whom you wish to run the peer assessment activity.

Note: The message ‘Exists’ or ‘Conflict’ means that the id (Identification) code has already been identified within your institution, or a previous Group you have uploaded. Carry on!

Check class and team membership

At this point you are unable to confirm the team membership of your team class. You must first launch a peer assessment activity selecting (one of) the Group Codes that existed within the original Participants CSV.

Quick links and related information

FAQ - I’m having problems importing my participants csv

FAQ: How do I view a list of the participants (team members) in the group I uploaded?

FAQ: How do I view or change the participants (team members) in a group I uploaded?

FAQ: How do I correct the Team Composition in a running peer assessment activity?

FAQ: Can I adjust the Start Date or Due Date for a running activity?

In short, No! Please check carefully your Start Date and Due Dates before you Create Activity.

3. Manage the Peer Assessment Activity

3.1 Action responses to warnings

Active Warnings show when you need to take action to remedy an issue during execution of the peer assessment activity.

In the following example, one member of Team Brazilia has completed the assessment of their four team members. Consequently, a warning is generated for Team Brazilia that the number of responses from the team is insufficient for presenting valid results. In contrast, all four team members of Team Kublas have completed the assessment.

The warnings displayed in this case are

Click through the warning to gain advice on how to remedy the situation. For example, you can remind the students to complete the survey. Emails are automatically generated and sent on your behalf to all or selected students.

Quick links and related information

FAQ: What is the content of emails sent by Peer Assess Pro?

FAQ - What emails have been sent by the platform?

Adjusting team composition

Upon commencing the peer assessment survey, team members are asked first to confirm that the team members identified or their team are correct. If not, the student initiates a request notification to the teacher to readjust their team’s membership.

Once the peer assessment activity has been launched, you can modify the team composition as per the following FAQ.

FAQ: How do I correct the Team Composition in a running peer assessment activity?

Note: Changes to a Xorro Group have NO EFFECT on current Team Composition

Changes to a Xorro Group will have NO EFFECT on a currently running activity, unless you Finalise then Abandon the activity. Then re-launch a new activity with the revised Group. This is an extreme response, and should not generally be required, if you follow the previous FAQ.

Quick links and related information

FAQ: How do I take action on the Warnings presented in the Peer Assess Pro™ Teacher’s Dashboard?’ What if I ignore the Warnings?

3.2 Automated and manual notifications

Students who have NOT completed the survey are sent an email reminder 72 hours, 24 hours and 12 hours before the Due Date.

Similarly, if a student is required to resubmit a response because a team has been reconstituted, an automatic reminder is sent.

Quick links and related information

FAQ: What is the content of emails sent by Peer Assess Pro?

FAQ - What emails have been sent by the platform?

3.3 Enter Team Results

The Team Results for each team must be entered should you intend to select any of these methods to calculate the Personal Result.

Indexed Personal Result
Normalised Personal Result
Rank Based Personal Result

After you have entered or revised your Team Results, communicate the Personal Results to your class using Publish or ‘Update’ button.

Team Results are not used to calculate:

Peer Assessed Score (Formerly Team Based Learning Score)
Peer Assessed Index (Formerly Indexed Team Based Learning Score)

Upon entering Team Results, the Peer Assess Pro platform selects automatically the Normalised Personal Result (NPR) method for calculating participants’ Personal result. A Scale Factor of 1.0 is selected.

3.4 Select the Personal Result Calculation Method

The Personal Result Calculation Method calculates the Personal Result you will award to each team member.

When you first enter Team Results, the Peer Assess Pro platform selects automatically the Normalised Personal Result (NPR) method for calculating participants’ Personal result. A Scale Factor of 1.0 is selected.

To adjust the Personal Result Calculation Method and/or adjust the Scale Factor

Select ‘Personal Result Method’ from Available Actions
Select the Calculation Method from the dropdown menu
Confirm ‘Recalculate’

Quick links and related information

FAQ: How do I decide which Personal Result method to apply in my peer assessment activity?

3.5 Review class, team, and individual statistics

You can explore progress and final results at the class, team, and individual level.

Review Class Results

In the Class Results, select a Bucket Range to identify the specific students lying within the range of a histogram bar chart.

Before reviewing results, see:

FAQ: When, why, and how do I ‘Recalculate Results’?

Example class statistics

In any of the tables, you may

Sort the table by clicking on the header for each column in the table
Select and view a personal snapshot of the peer feedback report that will be made available to a specific student when you Publish or Update the results.

Good practice hint: How to identify at risk students

Select Peer Assessed Score (PA Score) as the statistic for showing in Class Results
Sort the table ascending by PA Score
Interrogate the Individual Personal Snapshots of the lowest rating students.

The Individual Personal Snapshot

The Individual Personal Snapshot enables you to view all data related to one student. The Student View version of the Personal Snapshot shows exactlt the report the student will receive when the teacher Publishes the results of the current Peer Assess Pro activity.

However, the teacher may wish to view how the results will appear to students BEFORE they are Published. Consequently, there are four possible views of an Individual Personal Snapshot. They are variations on the following example. The four views are explained later.

Teacher’s Live View.
Student’ Live View
Student’s Published View.
Teacher’s Published View.

Example Individual Personal Snapshot (1 of 3)

Example Individual Personal Snapshot (2 of 3)

Example Individual Personal Snapshot (3 of 3)

Four possible views of the Individual Personal Snapshot

Note there are four possible views of an Individual Personal Snapshot.

Teacher’s Live View. Shows the feedback the student would view once the current (live) results are Published or Updated. Furthermore, this snapshot includes qualitative feedback in a transparent form where the teacher can view specifically

‘who said what’ by team member name
the individual ratings of the student by their team members
all Active Warnings pertaining to that student and their team.

Student’ Live View. The view not yet made available to the student, but what the student would view once the current results are Published or Updated. This snapshot includes qualitative feedback ‘who said what’ in anonymised form, just as the student would see the report.
Student’s Published View. The view you may have already Published to the Student, and available for their view. This snapshot includes qualitative feedback ‘who said what’ in anonymised form, just as the student would see the report
Teacher’s Published View. This view is similar to the view that is Published to the Student, and available for their view. Additionally, like the Teacher’s Live View, this snapshot shows ‘who said what’ and ‘who rated who’.

If the view is not yet Published, the student will see this remark.

Results unpublished

The same message will be also be displayed if the team is not a valid assessed team, even if the results have been Published to the class as a whole.

Team Statistics

Select an individual team to probe the results of its team members. Sort by Peer Assessed Score or Index of Realistic Self Assessment. Then you can quickly review the Individual Personal Snapshot of each team member as part of your diagnosis to identify ‘star performers’ , ‘at risk’ team members, and those with outlier degrees of over confidence or underconfidence.

Example Team Statistics

Qualitative Feedback

(To come)

Teacher’s Feedback

(To come)

Advanced Statistics

There are many advanced statistics and charts you can view. Furthermore, from ‘Available Actions’ you can Download Full Statistics to conduct more detailed investigations beyond the scope of what we have conceived.

Quick links and related information

FAQ: How is the Index of Realistic Self Assessment (IRSA) calculated?

3.6 Publish provisional Personal Results to team members

Results of the peer assessment are hidden from team members until you initiate Publish Survey on the Peer Assess Pro Teacher’s dashboard.

Before Publishing, see:

FAQ: When, why, and how do I ‘Recalculate Results’?

Unpublished status

Published status

Select ‘Refresh’ to ensure recent assessment submissions collected by Xorro are incorporated into the Peer Assess Pro database.
Select Recalculate to ensure changes to Team Results are incorporated into calculations of students’ results.
Select Publish Survey

The foregoing ‘Refresh and Recalculate’ steps provide you with the opportunity to quality review results before publishing and republishing personal results and qualitative peer feedback comments. In short, as the peer assessment activity progresses towards the due date, results ARE NOT automatically updated and made available for viewing by the students.

Take Care! Once an activity is Published, the results can never be unpublished. However, you may re-publish results if new responses are submitted and/or you make adjustments to Team Results, Team Composition, etc. To reiterate, even if interim results have been published to students, as the peer assessment activity continues to progress towards the due date, results ARE NOT automatically updated and made available for viewing by the students.

Results hidden when insufficient responses

Results will be hidden from the teacher and ALL team member in teams where less than one-half of team members have submitted the peer assessment. Peer assessment results are possibly not valid and representative at this stage of the survey activity processing. For small teams, at least 3 team members must have submitted a response. That is, team sizes of 3, 4, 5 and 6 team members require at least three team members to have peer assessed each other. A team of 7 or 8 requires a minimum of 4 responses. Team members who have already submitted a response will ALSO be advised their results are hidden until more of their team members have submitted responses.

Quick links and related information

FAQ: How do students know where and when to complete the peer assessment activity then review their results?

FAQ: How do I view and experience what the students experience?

4. Finalise the peer assessment activity

4.1 Why Finalise?

Survey responses from Team Members are received and available for incorporation into the Peer Assessment activity UNTIL the you explicitly Finalise the Survey. Even responses submitted after the Due Date announced to students, at the launch of the Activity, will be available for incorporation UNTIL the survey is Finalised deliberately by the Teacher. Until Finalisation, you can request a student to reconsider. They will then optionally resubmit their responses.

4.2 Publish Finalised Results to students

Check, resolve, or ignore Active Warnings.
Refresh and Recalculate.
Check again before Finalising publishing
Publish Survey to make available latest, final results to team members.

4.3 Download Teacher’s Gradebook of Results

From the Peer Assess Pro Teacher’s Dashboard, select either

Download Statistics - for a summary of the gradebook, by student name, id, Personal Result
Download Full Statistics - for a complete breakdown of the components scores for each student, all personal result possibilities, and IRSA.
Conduct a quick check of these results before you complete the next step, Finalise!

Example Gradebook Summary Statistics

Example Gradebook Full Statistics

4.4 Finalise the Activity … irrevocably!

CRITICAL WARNING: From this point the Teacher can make NO FURTHER adjustments to Team Results, Team Composition, or Calculation Method. No further responses will be accepted.
Select ‘Finalise Survey’.
Students will be advised that their finalised results are available to review from their personal Xorro Dashboard
Results are available for the student’s view for upto two weeks following Finalisation.
Two weeks after Finalisation the state of the peer assessment activity changes from being a RUNNING ACTIVITY to an expired RECENT ACTIVITY. On the Xorro ACTIVITIES page, the ACTIVITY is now listed in RECENT ACTIVITIES. Students can no longer view the results of an (expired) RECENT ACTIVITY.
The responses and responses of the activity are archived permanently in the Teacher’s Recent Activities schedule on Xorro.
The Teacher may continue to download the Gradebook results and Qualitative reports after Finalisation and Expiration.

Quick links and related information

FAQ: How do students know where and when to complete the peer assessment activity then review their results?

FREQUENTLY ASKED QUESTIONS

Quick Link Map

FAQs for teachers

Quickstart Guide for teachers

The peer assessment survey

Launch peer assessment activity

Manage the peer assessment activity

Responding to Active Warnings

Definitions, calculations, and examples

Miscellaneous

The peer assessment survey

FAQ - What is the benefit of a standardized peer assessment rubric?

FAQ - When and how is the peer assessment conducted?

FAQ - What is the purpose of peer assessment?

FAQ - What questions are asked in the peer assessment survey?

FAQ - How do students know where and when to complete the peer assessment activity then review their results?

FAQ - How are peer assessment and personal results calculated and defined mathematically?

FAQ - Is the self-assessment used to calculate Peer Assessed Score?

Login and orientation

FAQ - Give me a quick overview of how to launch a Peer Assess Pro™ activity through Xorro

FAQ - How do I navigate the PARTICIPANTS page for Peer Assess Pro?

FAQ - How do I view and experience what the students experience?

Launch peer assessment activity

FAQ - I’m having problems importing my participants csv

FAQ - How do I create a CSV file from a Google Sheet?

FAQ - Can I create a peer assessment activity without having all my teams correctly identified by team name and/or team membership?

FAQ - How do I correct the participants (team members) in a group already uploaded to Xorro?

FAQ - How do students know where and when to complete the peer assessment activity then review their results?

FAQ - Can I adjust the Start Date or Due Date for a running activity?

FAQ - What if a student mistakenly advises they are in an incorrect team?

Manage the peer assessment activity

FAQ - Can I adjust the Start Date or Due Date for a running activity?

FAQ - How do I correct the Team Composition in a running peer assessment activity?

FAQ - How do I resolve an unsynchronised team arrangement? (ACTIVE WARNING 0021)

FAQ - What if a student mistakenly advises they are in an incorrect team?

FAQ - What is the content of emails sent by Peer Assess Pro to Participants?

FAQ - What is a valid assessed team?

FAQ: How do I decide which Personal Result method to apply in my peer assessment activity?

FAQ - When, why, and how do I ‘Update and Recalculate Results’?

FAQ - How do I take action on the Warnings presented in the Peer Assess Pro™ Teacher’s Dashboard?’ What if I ignore the Warnings?

FAQ - What happens if [a student tries] to 'game' (fool? play? disrupt?) the peer assessment process?

FAQ - How do I advise a student who feels they have been unfairly treated?

FAQ - What emails have been sent by the platform? (Notifications History)

Responding to Active Warnings

FAQ - How do I take action on the Warnings presented in the Peer Assess Pro™ Teacher’s Dashboard?’ What if I ignore the Warnings?

FAQ: What is an adjusted teamset request? WARNING 0006

FAQ - How do I resolve an unsynchronised team arrangement? ACTIVE WARNING 0021

FAQ - What is the content of emails sent by Peer Assess Pro to Participants?

FAQ - What is a valid assessed team? WARNING 0022

FAQ - How do I fix an invalid, missing or failed email delivery? WARNING 0026

FAQ - What is a team with low psychological safety? WARNING 0034

FAQ - What is an unsafe team member? WARNING 0035

FAQ - What is an ‘at risk’ team member? WARNING 0036

FAQ - What is a mismatched self-assessment (IRSA)? WARNING 0040

FAQ - How is insignificant team agreement identified? WARNING 0041

FAQ: How is an outlier peer assessment rating identified? WARNING 0042

FAQ - What is an inactive team member? WARNING 0048

FAQ - What is a low quality team rating? WARNING 0050

FAQ - What is a low quality assessor rating? WARNING 0300

Definitions, calculations, and examples

FAQ - How are peer assessment and personal results calculated and defined mathematically?

FAQ - What is the benefit of a standardized peer assessment rubric?

FAQ - How is the Peer Assessed (PA) Score calculated?

FAQ - Is the self-assessment used to calculate Peer Assessed Score?

FAQ - How is the Peer Assessed Index (PA Index) calculated?

FAQ - How is the Indexed Personal Result (IPR) calculated?

FAQ - How is the Normalised Personal Result (NPR) calculated?

FAQ - How is the Rank Based Personal Result (RPR) calculated?

FAQ - How is Standard Peer Assessed Score (SPAS) calculated?

FAQ - What is Employability? How is it calculated?

FAQ - How is the Index of Realistic Self Assessment (IRSA) calculated?

FAQ - How do I decide which Personal Result method to apply in my peer assessment activity?

Miscellaneous

FAQ - I’m having problems importing my participants csv

FAQ - How do I contact people at Peer Assess Pro?

FAQ - Where may I view the most recent version of the user guides?

FAQ - Why are different terms used to display peer assessment results in the Xorro and previous Google versions of Peer Assess Pro™?

FAQ - What are the design objectives, key features, and benefits of the Peer Assess Pro development?

FAQs for team members

The purpose of peer assessment

Undertaking the peer assessment

Using peer assessment results for better performance

How peer assessment affects personal results

The purpose of peer assessment

FAQ - What is the purpose of peer assessment?

FAQ - How are peer assessment and personal results calculated and defined mathematically?

Undertaking the peer assessment

FAQ - What questions are asked in the peer assessment survey?

FAQ - I am unable to login. My login failed

FAQ - How do I login to my peer assessment Activity URL

FAQ - What if I mistakenly advise the survey I am in an incorrect team?

FAQ - When and how is the peer assessment conducted?

FAQ - How do I provide useful feedback to my team members?

FAQ - How do I know where and when to complete the peer assessment activity then review their results?

FAQ - How do I view and experience what the students experience?

FAQ - Is the self-assessment used to calculate Peer Assessed Score?

FAQ - What happens if I try to 'game' (fool? play? disrupt?) the peer assessment process?

Using the results from peer assessment for better performance

FAQ - How do I interpret the feedback results I've received from the peer assessment?

FAQ - How do I interpret measures of realistic self-assessment?

FAQ - What steps can I take to get a better personal result?

FAQ - Is the self-assessment used to calculate Peer Assessed Score?

FAQ - I don’t understand what my teammates are trying to tell me. How do I ask for better feedback?

FAQ - I believe I have been unfairly treated by the results of the peer assessment. How do I address my concern?

FAQ - What is a team with low psychological safety?

FAQ - What is an unsafe team member?

FAQ - What is an ‘at risk’ team member?

FAQ - What is Employability? How is it calculated?

How peer assessment affects personal results

FAQ - How are peer assessment and personal results calculated and defined mathematically?

FAQ - Is the self-assessment used to calculate Peer Assessed Score?

FAQ - What steps can I take to get a better personal result?

FAQ - What happens if I try to 'game' (fool? play? disrupt?) the peer assessment process?

FAQ - I believe I have been unfairly treated by the results of the peer assessment. How do I address my concern?

FAQ: What is the purpose of peer assessment?

Defining peer assessment

Peer assessment is an educational activity in which students judge the performance of their peers, typically their teammates. Peer assessment takes several forms including

Summative, where the peer assessment is used to determine a team member’s academic grade or personal result
Formative, where the peer assessment provides feedback that a team member can use to improve their contribution to the future work of a team in which they are working and gain a better personal result or grade.

Developmental feedback

The ability to give and receive constructive feedback is an essential skill for team members, leaders, and managers.

Consequently, your teacher has chosen to use Peer Assess Pro™ to help you provide developmental feedback to your team members, for both formative and/or summative purposes.

The goal of developmental feedback is to highlight both positive aspects of performance plus areas for performance improvement. The result of feedback is to increase both individual and team performance (Carr, Herman, Keldsen, Miller, & Wakefield, 2005).

Determination of course personal result

Additionally, your teacher may use the quantitative results calculated by Peer Assess Pro™ to determine your Personal Result for the team work conducted by your team. Your Personal Result may contribute to the final (summative) assessment grade you gain for the course in which Peer Assess Pro™ is applied.

In general, your Personal Result is calculated from two factors:

Team Result, the overall mark your team earns from its team project outputs, such as reports and presentations
Peer Assessed Score, a measure of the relative contribution your team members assess that you have made to the team’s delivered outputs, teamwork processes, and/or leadership of the team.

Criteria for peer assessment in Peer Assess Pro™

There are many possible criteria for assessing your contribution to your team’s work. Peer Assess Pro has chosen to place equal weight on two groups of factors based on a well-established instrument devised by Deacon Carr, Herman, Keldsen, Miller, & Wakefield (2005), Task Accomplishment, and Contribution to Leadership and team processes:

Task Accomplishment: Contributions you make directly to help the team accomplish its task, such as showing initiative, professionalism, contributing ideas, helping other team members learn relevant concepts, and making well-prepared on-time contributions to meetings and team deliverables
Leadership and Team Processes. Contributions such as focussing the team on priorities, encouraging and welcoming the contributions of others, managing conflict, and chairing meetings productively.

Peer Assess Pro assesses competencies valued by employers

The selection of the criteria used in the Peer Assess Pro is reinforced by the results from a recent survey that asked employers to rate the importance of several competencies they expected to see in new graduates from higher education. The figure shows teamwork, collaboration, professionalism, and oral communications rate amongst the most highly needed Career Readiness’ Competencies (CRCs) sought by employers. All these Career Readiness competencies rate at least as ‘Essential’, with Teamwork and Collaboration rating almost Absolutely Essential (National Association of Colleges and Employers, 2018).

Employers rate their essential need for Career Readiness Competencies

Source: National Association of Colleges and Employers (NACE). (2018). Figure 42, p. 33.

Quick links and related information

FAQ: What questions are asked in the peer assessment survey?

FAQ: How are peer assessment and personal results calculated and defined mathematically?

FAQ: How is the Peer Assessed (PA) Score calculated?

Video

Mellalieu, P. J. (2021, June 9). Why Peer Assessment? The key to improved group assignments. Better Feedback. Better Teams. https://www.peerassesspro.com/why-peer-assessment/

References

Deacon Carr, S., Herman, E. D., Keldsen, S. Z., Miller, J. G., & Wakefield, P. A. (2005). Peer feedback. In The Team Learning Assistant Workbook. New York: McGraw Hill Irwin.

National Association of Colleges and Employers (NACE). (2018). Job Outlook 2019. Bethlehem, PA. https://www.naceweb.org/

FAQ: When and how is the peer assessment conducted?

The best practice for conducting peer assessment in an academic course follows several stages.

Introduction. The teacher introduces the team activity and related course assignments
Peer assessment purpose. The teacher explains the role, purpose, and process of peer assessment
The team activity commences
A formative peer assessment is conducted using Peer Assess Pro™ early or mid-way through the team activity.
Team members receive formative feedback generated by Peer Assess Pro™ that indicates their provisional peer assessment rating and (optionally) their indicative end of class personal result. More importantly, they receive qualitative information that provides guidance on what behaviors are required to improve their contribution towards the results the team is seeking.
The team activity continues towards its conclusion. Team members confirm informally with each other that they are correctly applying the more productive behaviours identified through the formative peer assessment.
The team activity concludes.
The summative peer assessment using Peer Assess Pro™ is conducted at the conclusion of the team activity and/or before the conclusion of the course.

Formative assessment: optional but valuable

The midpoint formative peer assessment is an optional element of peer assessment within the classroom. As a minimum, the formative peer assessment gives the team members experience of the Peer Assess Pro™ mechanism including the questions that will be used to conduct the final, summative peer assessment.

More importantly, the midpoint formative assessment helps ensure that team members have the opportunity to respond proactively to the peer feedback they receive immediately the peer assessment activity concludes. Through undertaking appropriate corrective action mid-way through the course, team members have the opportunity to raise their peer assessment rating, their team’s results, and, therefore, their end of course personal results.

No surprises!

The intention of formative assessment is that, ideally, a team member should face no surprises when they receive their final personal result and peer assessment feedback at the conclusion of the course. For instance, a free-riders should receive clear feedback that the rest of their team observes they are free-riding. Consequently, the free-rider should learn in a timely manner that they will be penalised at the concluding summative assessment unless they remediate their behaviour. It is equally important that an overachieving student who does most of the work is given timely feedback that they need to learn to involve and engage the other team members in the team’s planning and execution of tasks. The Peer Assess Pro™ survey specifically targets these aspects of leadership and team process contributions. This particular style of overachieving student should be identified through the peer assessment ratings they receive.

To minimise the risk of surprises, it is important, therefore, that the peer assessment you provide to your team members at the midpoint of a team activity is

Fair, accurate, and honest
Supported by qualitative feedback that supports the peer assessment ratings you provided in the Peer Assess Pro™ survey
Reinforced by specific, actionable advice to the assessed team member about what behaviours they must undertake to improve their contribution to either the team’s outputs, or the team’s leadership and group processes.

Quick links and related information

FAQ: What questions are asked in the peer assessment survey?

FAQ: How do I provide useful feedback to my team members?

FAQ: How do students know where and when to complete the peer assessment activity then review their results?

FAQ: How do I view and experience what the students experience?

FAQ: How do I interpret the feedback results I've received from the peer assessment?

FAQ: How are peer assessment and personal results calculated and defined mathematically?

FAQ: Is the self-assessment used to calculate Peer Assessed Score?

FAQ: How do I provide useful feedback to my team members?

It is essential that the peer assessment a team member provides to their team members through peer assessment is:

Fair, accurate, and honest
Supported by qualitative feedback that supports the peer assessment ratings provided in the Peer Assess Pro™ survey
Reinforced by specific, actionable advice to the assessed team member about what behaviours they must undertake to improve their contribution to either the team’s outputs, or the team’s leadership and group processes.

Ohland et al (2012) provide a table of Behaviorally Anchored Ratings covering high and low contributions to team effectiveness. The table provides some guidance to team members about how they might give accurate, effective, and productive feedback to their team members through peer assessment.

Examples of high and low contributions to team effectiveness
HIGH	CONTRIBUTION	LOW
Does more or higher quality work than expected. Makes important contributions that improve the team's work. Helps to complete the work of teammates who are having difficulty. Completes a fair share of the team's work with acceptable quality.	CONTRIBUTION	Does not do a fair share of the team's work. Delivers sloppy or incomplete work. Misses deadlines. Is late, unprepared, or absent for team meetings. Does not assist teammates. Quits if the work becomes difficult.
Asks for and shows an interest in teammates' ideas and contributions. Improves communication among teammates. Provides encouragement or enthusiasm to the team. Asks teammates for feedback and uses their suggestions to improve. Listens to teammates and respects their contributions. Communicates clearly. Shares information with teammates. Participates fully in team activities. Respects and responds to feedback from teammates.	INTERACTION	Interrupts, ignores, bosses, or makes fun of teammates. Takes actions that affect teammates without their input. Does not share information. Complains, makes excuses, or does not interact with teammates. Accepts no help or advice.
Watches conditions affecting the team and monitors the team's progress. Makes sure that teammates are making appropriate progress. Gives teammates specific, timely, and constructive feedback. Notices changes that influence the team's success. Knows what everyone on the team should be doing and notices problems. Alerts teammates or suggests solutions when the team's success is threatened.	KEEPING FOCUS	Is unaware of whether the team is meeting its goals. Does not pay attention to teammates' progress. Avoids discussing team problems, even when they are obvious.
Demonstrates the knowledge, skills, and abilities to do excellent work. Acquires new knowledge or skills to improve the team's performance. Able to perform the role of any team member if necessary. Has sufficient knowledge, skills, and abilities to contribute to the team's work. Acquires knowledge or skills needed to meet requirements. Able to perform some of the tasks normally done by other team members.	CAPABLE	Missing basic qualifications needed to be a member of the team. Unable to perform any of the duties of other team members. Unable or unwilling to develop knowledge or skills to contribute to the team.
Source: Ohland et al., (2012)
Adapted by Mellalieu (2017) from Ohland, M. W., Loughry, M. L., Woehr, D. J., Bullard, L. G., Felder, R. M., Finelli, C. J., … Schmucker, D. G. (2012). APPENDIX B: Behaviorally Anchored Rating Scale (BARS) Version, from Comprehensive Assessment of Team Member Effectiveness. Academy of Management Learning & Education, 11(4), 609–630. Retrieved from http://amle.aom.org/content/11/4/609.short

Quick links and related information

FAQ: I believe I have been unfairly treated by the results of the peer assessment. How do I address my concern?

For teachers: How do I advise a student who feels they have been unfairly treated?

Symptoms of an unfair assessment

Here are some symptoms that you may have been treated unfairly by one or more teammates in their peer assessment of you:

The qualitative and development feedback provided from your team members provides no obvious support for the ratings they gave you
You honestly have no idea what you could have done to match or exceed the contributions made by other team members to your team’s output and/or process or working together
Your self-assessment differs markedly from the peer assessment score awarded by your team members. This divergence is highlighted by a low Index of Realistic Self-Assessment (IRSA), that is, an IRSA below 70.

Steps to address an unfair peer assessment

If you believe you may have been unfairly treated, these are the steps you should pursue, in this order of action

Ensure that you understand the results of your peer assessment, and how the results are calculated. See the Quick Links below for suggestions.
Email then arrange to discuss your concerns with your teacher. Your teacher can check the peer assessments provided by your team members. For example, perhaps one or more extreme peer assessments have affected negatively your result. The teacher can explore that possibility with you and the team member concerned. You should be able to present evidence to the teacher of the work you agreed to complete, your record of attending meetings, the work you produced, and relevant communications from your team members expressing their satisfaction or otherwise of your contributions to team outputs and/or working processes.
Meet with your team members to ensure you fully understand WHY they have awarded you the ratings they gave. In particular, ensure you understand accurately what you must do to achieve a better peer assessment rating in the future. In addition, when you meet your teacher in Step 2, you might request the teacher’s attendance at this meeting as a meeting observer or facilitator ‘to keep the peace’.
If there remains a dispute between your self-assessment and your team’s assessment, the teacher MAY with discretion request that one or more of your team members resubmit their peer assessment. This step can only be undertaken if the peer assessment activity is not finalised.
If you continue to dispute your peer assessment and/or personal result then you should pursue your institution’s policy for appealing an assessment result. This policy is usually mentioned in your course syllabus, assignment specification, programme overview, and/or learning management system.

A note on appealing a peer assessment result

An appeal against a peer assessment result is likely to fail if one or more of the following circumstances have prevailed:

You failed to pursue in a timely manner the previously-described steps to address your perception of an unfair result. Do not wait until you receive your end of class results. Address your concern promptly when you receive your Peer Assess Pro peer assessment results.
The teacher conducted both a formative and summative peer assessment. There is evidence that you received a poor formative (first) peer assessment. You failed subsequently to respond proactively and productively to the poor formative peer assessment. In other words, you had every opportunity to understand how the peer assessment process would affect your personal result, and what you required to achieve a better result in the summative, end of class peer assessment.
You have received poor peer assessment results from other classes and/or teachers. There is little evidence you have responded productively to those previous poor peer assessments.
There is strong consistency between the qualitative feedback, developmental feedback and the quantitative peer ratings that comprise your Peer Assessed Score.
You have attempted to coerce your team members into awarding you a favourable rating. The act of coercion may be regarded by your institution as academic dishonesty, bullying and/or harrassment. Your institution will have vigorous policies that prosecute seriously any behavior such as coercion, bullying, and harassment. The consequences typically range from a failure for the assignment or course through to exclusion from the programme or institution.
The formative peer assessment across the entire team was of low quality AND the Team Result awarded by the teacher is low. A low quality peer assessment by the entire team is evidenced where everyone rated each other with a Peer Assessed Score of between 90 to 100, or over a narrow range. This is self-inflicted injury on behalf of the entire team for failing to take productive advantage of the peer assessment process as a means to improve their team-work processes.

Prevention is better than cure

Take these steps to avoid a mismatch between the peer assessment result you expect, and the result you receive.

Ensure you understand how your Peer Assessment Score and your teacher-awarded Team Result combine to produce the personal result you will receive for your contribution to team outputs and teamwork processes.
Provide honest, accurate and constructive feedback to your team members.
Seek to understand from your peer feedback results what behaviours and attitudes you should adopt to add value to your team’s process and results.
Implement the changes required.
Proactively check with your team members that you are on track with a more productive approach.

Quick links and related information

How peer assessment affects personal results

FAQ: How are peer assessment and personal results calculated and defined mathematically?

FAQ: Is the self-assessment used to calculate Peer Assessed Score?

FAQ: What happens if I try to 'game' (fool? play? disrupt?) the peer assessment process?

Using the results from peer assessment for better performance

FAQ: How do I interpret the feedback results I've received from the peer assessment?

FAQ: I don’t understand what my teammates are trying to tell me. How do I ask for better feedback?

FAQ: How do I interpret measures of realistic self-assessment?

FAQ: What steps can I take to get a better personal result?

FAQ: How do I interpret the feedback results I've received from the peer assessment?

(To be published)

Quick links and related information

FAQ: How do I interpret measures of realistic self-assessment?

FAQ: I don’t understand what my teammates are trying to tell me. How do I ask for better feedback?

Begin by viewing this video. Watch especially for the question that is introduced soon after minute 15 by Harvard University professor Sheila Heen.

Heen, S. (2015). How to use others’ feedback to learn and grow. TEDx. Retrieved from https://www.youtube.com/watch?v=FQNbaKkYk_Q

As Heen and Stone observe

“Feedback is less likely to set off your emotional triggers if you request it and direct it. So donʼt wait until your annual performance review. Find opportunities to get bite-size pieces of coaching from a variety of people throughout the year. Donʼt invite criticism with a big, unfocused question like “Do you have any feedback for me?” Make the process more manageable by asking a colleague, a boss, or a direct report,

“Whatʼs one thing you see me doing (or failing to do) that holds me back?”

That person may name the first behavior that comes to mind or the most important one on his or her list. Either way, youʼll get concrete information and can tease out more specifics at your own pace.” (Heen & Stone, 2014)

Quick links and related information

Heen, S., & Stone, D. (2014). Find the Coaching in Criticism. Harvard Business Review, 9. Retrieved from https://medschool.duke.edu/sites/medschool.duke.edu/files/field/attachments/find-the-coaching-in-criticism.pdf

FAQ: What steps can I take to get a better personal result?

Your Personal Result is determined from a combination of your Team Result and your Peer Assessed Score. Consequently, to raise your Personal Result you need to apply balanced effort to raising both these contributing factors.

Raise your Team Result

Typically, your Team Result is earned from its assignment outputs, such as a report, and/or a presentation. Consequently, the grade for the Team Result is determined by the teacher, based on the rubric (marking guideline) they apply to assess your team’s outputs. Ensure you understand the assignment elements and how each will be assessed. Seek out exemplars of good practice. Pursue the guidance found in:

Mellalieu, P. (2013, March 15). Creating The A Plus Assignment: A Project Management Approach (Audio). Innovation & chaos ... in search of optimality website: http://pogus.tumblr.com/post/45403052813/this-audio-tutorial-helps-you-plan-out-the-time

Use your institution’s academic support services

In addition to your teacher and their assistant tutors, your academic institution will offer personal and group coaching to guide you on the specific success factors related to the type of assignment you are pursuing. Schedule appointments to make use of these support facilities early in your project. Locate the online resources these coaching support services have curated for your guidance.

Raise your Peer Assessed Score

Group and team projects present special challenges of coordination, motivation, communication and and leadership. These challenges are normal! Furthermore, an essential part of your job as team member is to overcome proactively these challenges as part of your academic learning journey.

As you overcome these challenges you will achieve several benefits directly instrumental in raising your Personal Result:

Your Team Result will increase
Your team members will recognise that you have contributed to the team’s success in raising its Team Result
Your team members, therefore, will raise their peer assessment of your contribution. So, therefore,
Your Peer Assessed Score will rise.

You will also develop team work and leadership competencies that will both raise your future employability, and your effectiveness in future teamwork, as discussed in:

FAQ: What is the purpose of peer assessment?

How do I address proactively the challenges of team work?

Whilst there are many resources to help address the challenges of team work in academic settings, we suggest you familiarise yourself with these resources early in your team project. Since “Any fool can learn from their own mistakes. It takes genius to learn from the mistakes of others” (Einstein), be proactive rather than foolish in learning effective team working skills from:

Turner, K., Ireland, L., Krenus, B., & Pointon, L. (2011). Collaborative learning: Working in groups. In Essential Academic Skills (2nd ed., pp. 193–217).

Carr, S. D., Herman, E. D., Keldsen, S. Z., Miller, J. G., & Wakefield, P. A. (2005). The Team Learning Assistant Workbook.

Learning constructively from mid-course peer assessment feedback

Good practice peer assessment management by your teacher will provide you with two opportunities for peer assessment and peer feedback through your course, formative and summative.

Your first, mid-course, formative assessment provides you with early advice about your strengths and opportunities for development as perceived by your team members. Make use of this formative feedback at the earliest opportunity as you proceed towards the conclusion of your team work, and your final, summative peer assessment. Usually, this final, summative assessment is where you earn the significant contribution to your course grade from the Personal Result earned from your Peer Assessed Score awarded by your team members.

Consequently, take proactive action following the mid-course formative assessment through referring to:

FAQ: How do I interpret the feedback results I've received from the peer assessment?

Maybe you don’t understand or don’t agree with the feedback your teammates are providing. In that case, refer to

FAQ: I don’t understand what my teammates are trying to tell me. How do I ask for better feedback?

Quick links and related information

The purpose of peer assessment

FAQ: What is the purpose of peer assessment?

Undertaking the peer assessment

Using peer assessment results for better performance

How peer assessment affects personal results

FAQ: How are peer assessment and personal results calculated and defined mathematically?

FAQ: What steps can I take to get a better personal result?

FAQ: What happens if I try to 'game' (fool? play? disrupt?) the peer assessment process?

FAQ: Is the self-assessment used to calculate Peer Assessed Score?

FAQ: What happens if I try to 'game' (fool? play? disrupt?) the peer assessment process?

What happens if a team member attempts to 'game' the peer assessment process?

The designers of Peer Assess Pro have many decades’ experience working with students. We know the tricks that students attempt to play with peer assessment. We have anticipated the tricks, so Peer Assess Pro warns the teacher that a trick may be being played. Furthermore, the teacher receives highly specific and student individualized information about each incident. The teacher may then undertake overt or covert action to address the issue to which they have been alerted. For example, the trick-playing student or team may then receive a request to reconsider and resubmit their peer assessment. In more extreme incidents, the student or team may receive an invitation to visit the teacher for a counselling consultation.

The tricks we know!

Here follow a few of the ‘tricks’ that Peer Assess Pro identifies and warns the teacher about during the survey process. Examples follow later.

Low quality assessor rating: An assessor may have engaged unconstructively with peer assessment. Also known as “I can’t be bothered with this. I’ll give everyone the same rating”
Low quality team rating: A team may have engaged unconstructively with peer assessment. Also known as “If we give everyone a high mark, we’ll get a better personal result won’t we?”
Outlier individual rating: A team member has assessed another team member very differently than the other team members. Also known as “This is my friend, I’ll rate her well” or “We’ve never got on well together. Now this is my chance for revenge!”
Mismatched self-assessment: A team member's self-assessment is materially different from the peer assessment given by their team. Also known as “If I give myself a high rating, I’ll get a better personal result won’t I?”
At risk team member: A team member has been rated amongst the lowest in class. Also known as “Darn! I thought I could hide under the radar”

Examples: Highly specific and individualized information

Here are some examples of the highly specific and individualized Active Warnings a teacher receives about each incident.

1. Low quality assessor rating

Madison may have engaged unconstructively with peer assessment. High Peer Assessment Scores awarded. Average 100 and low range 3. Team Alpha

Ben may have engaged unconstructively with peer assessment. High Peer Assessment Scores awarded. Average 86 and low range 5. Team Bravo

This message warns the teacher that the team member has given everyone a near perfect Peer Assessed Score or a similar score (narrow range). Practically, from the student’s point of view they are ‘wasting their votes’. If everyone is scored with the same or similar score then students who have contributed substantially to the team’s result will not be adequately recompensed. Furthermore, if EVERY team member pursued this same approach, then every team member would be awarded the Team Result. In this case, the team member just looks stupid in the eyes of the teacher. Furthermore, the team member fails to gain practice at being a leader where giving accurate assessments of team members’ contributions is a valued management competency.

2. Low quality team rating

Team Bravo may have engaged unconstructively with peer assessment. High Peer Assessment Scores awarded. Average 98 and low range 8.

Team Echo may have engaged unconstructively with peer assessment. High Peer Assessment Scores awarded. Average 94 and low range 10.

These messages warns the teacher that the team collectively may have arranged to give everyone a near perfect Peer Assessed Score or the same score. Practically, from the students’ point of view, this trick is a waste of time. If everyone is scored with the same score, or a perfect Peer Assessed Score of 100, then every team member will be awarded the Team Result … which is usually not 100. The team members just look stupid in the eyes of the teacher. Furthermore, they may not receive useful qualitative feedback and ratings that help guide focussed development of their future productivity in team assignments and their future professional work in teams.

3. Outlier individual rating

The warning highlights a situation where the team members appear to be inconsistent in rating high, medium and low contributors to the team’s process and results.

This example is a symptom there maybe some disruptive team dynamics or bullying within the team.

Harrison Ford assessed Steven Spielberg awarding a PA Subscore of 38. Compared with the average rating by the other team members of 70 this subscore DEPRESSED the PA Score to 64 by 7 PA Score Units. Team Alpha

This message warns the teacher there maybe some favouritism between friends or allies.

Donald Trump assessed Vladimir Putin awarding a PA Subscore of 90. Compared with the average rating by the other team members of 57 this subscore RAISED the PA Score to 64 by 7 PA Score Units. Team Charlie

4. Mismatched self-assessment

Anna self-assessment of 68 is OVERCONFIDENT compared with the peer assessment of 34 given by others in team Charlie. IRSA = 51

This message warns the teacher that the team member has a very much higher opinion of their performance than is evidenced by the rating provided by their peers. The teacher may request an interview with the student to explore the reasons for this divergence, and how the student can develop a more realistic self-assessment.

Alternatively, the team member may be being scape-goated by the remainder of the team, and that possibility will be discussed with the team member for whom this warning is raised.

5. At risk team member

Anna has been rated amongst the lowest in class. Low Recommendation 2.3 and/or low Peer Assessment Score 34. Team Alpha

This message warns the teacher that the team member is rated very poorly when compared with most of the class. It’s often a symptom of little or no attendance or contribution by the team member, which the teacher will verify through examining the qualitative feedback provided by the team members. Again, the teacher may request an interview with the student to explore the reasons for this divergence, and how the student can develop a more realistic self-assessment.

Example: Better feedback. Better teams

Examine the following teacher’s dashboard graphic revealing a real class that undertook a peer assessment.

Which teams would raise the Active Warning: Low quality team rating?
Which teams tend to have a higher team result?
Which teams have worked most productively as a team?

Teachers dashboard: visible identification of teams with low quality team rating

Which teams will raise the Active Warning: Low quality team rating?

Team 1, 14, 13, 2, 7, 5, and 11. Over ½ of the teams in the class!

Observation: This class was poorly briefed on how to make the best use of peer assessment and feedback. With a better briefing, less than 10% of teams will raise this warning.

Which teams tend to have a higher team result?

The lower Team Results are associated with teams that had a low quality team rating. Apart from Team 15, all teams with an adequate quality team rating had a Team Result equal to or greater than the Class median Team Result of 73.3. For example, Team 10 (Team Result 90) through to Team 4 (Team Result 76.7) according to the sort by Range in the foregoing table.

Which teams have worked most productively as a team?

Team 10, with a Team Result of 90 is clearly a high performing team. The moderately low Range of PA Scores (10) across the team suggests IN THIS CASE that everyone contributed relatively equally and effortfully towards a great Team Result. Reminder: The Team Result is awarded by the teacher: it is independent of the Peer Assessed Scores of the team.

However, Team 3 is also a good candidate for being a fair and productive team. They engaged honestly with peer assessment, awarding a high spread of Peer Assessed Scores (Range 18.8) an a team average PA Score (78.3). This team average was not outrageously high, in contrast to teams 1 (100!), 14 (100), 13, 2, 7, 5. Furthermore, Team 3 earned the class median Team Result of 73.3, which appears then allocated according to the peer assessed contribution of the team members. This fair distribution is illustrated in the following graph and table. Team Member Charlie earned the highest Personal Result of 81, whilst Able earned 65.3. Similar reasoning applies to Team 6 to a slightly lesser degree, since the Range is not so wide.

Note from the following graph how teams 14, 5, 13, 1, 2 and 7 are again glaringly identified in the Teachers Dashboard as outlier teams poorly engaged with the peer assessment process: the low vertical spread in the graph. This low vertical spread in the Personal Result (NPR in this case) derives from the low range of Peer Assessed Scores across each team.

With this admittedly small case size, we advance the proposition that ‘Better feedback leads to better teams’. And/Or ‘Better teams give better feedback!’. In conclusion, let’s say Better feedback. Better teams.

Teachers dashboard: a fairly productive team

Quick links and related information

FAQ: What is the purpose of peer assessment?

FAQ: How do I provide useful feedback to my team members?

FAQ: How do I interpret the feedback results I've received from the peer assessment?

FAQ: I don’t understand what my teammates are trying to tell me. How do I ask for better feedback?

FAQ: I believe I have been unfairly treated by the results of the peer assessment. How do I address my concern?

Active Warnings, thresholds parameters, and program logic

The following section explains how the teacher should respond to the Active Warnings displayed on their dashboard. The thresholds parameters and program logic for raising the warnings are also provided.

Responding to Active Warnings

FAQ: Give me a quick overview of how to launch a Peer Assess Pro™ activity through Xorro

Register for a free Xorro Teacher's Account, at https://www.peerassesspro.com/first-time-registration/
Login to Xorro with your Xorro registration details - http://qf.xorro.com/
Create your class list as a csv file containing your class list organised into teams. Use this example format https://www.xorro.com/wp-content/uploads/2020/10/Participant_CSV_Sample.csv
Launch a peer assessment activity. You will be presented with the option to import your CSV class list of teams - https://qf.xorro.com/pap/launches/new
Manage your peer assessment activity by returning to - https://qf.xorro.com/activities/running

If this is your first time using Peer Assess Pro, we recommend strongly that you glance briefly our Frequently Asked Questions so you are prepared to answer your own and your students' concerns - https://www.peerassesspro.com/frequently-asked-questions-2/

Download a pdf of the Quickstart Guide and this Reference Guide here - http://tinyurl.com/papRefPdf

View a quick video overview demonstration of the whole Peer Assess Pro system

Contact

Patrick Dodd - https://www.peerassesspro.com/contact/

Quick links and related information

View the web Quickstart Guide at tinyurl.com/pdfQuickWeb

View our comprehensive online and eBook introduction Get Started with Peer Assess Pro

FAQ: How do I contact people at Peer Assess Pro?

FAQ: Where may I view the most recent version of the User Guide?

FAQ: What are the design objectives, key features, and benefits of the Peer Assess Pro development?

Design objectives

Our overall objectives for Peer Assess Pro™ are

Streamline students’ experience when completing the peer assessment
Automate for the teacher many functions for launching and managing a peer assessment
Reduce the chance of accidents and errors in use. This includes streamlining the teacher’s administration of warning alerts about exceptional situations
Provide the teacher with a structured approach to revealing results, probing specific teams’ and students’ performance and responses, and options for detailed exploratory data analysis.

Benefits for students

A student will be able to rate all their team members in one combined peer assessment activity.
A students may more easily rate comparatively the performance of their team members including themself
Students are provided with the facility to advise the teacher directly if they believe their team composition has been specified incorrectly by the teacher
Students can update their assessment response at any time in response to a directive from their teacher. The last submitted response alone will be used in the peer assessment calculations
Students view their assessment survey, peer assessment results, and personal result (grade) by returning to a single URL communicated to them automatically by the Peer Assess Pro™ Teacher’s dashboard. Results are available for students’ view for no more than two weeks after the teacher has finalised the Peer Assess Pro™ Assessment Activity
Optionally, a student may view all their previous peer assessment activities through the Xorro Plus Dashboard. Xorro Plus enables the student to provide authoritative evidence supporting how their teamwork capabilities have been established and, hopefully, improved over successive participation in peer assessed team projects throughout their academic studies.

Benefits for teachers

A one stop shop for launching, managing, and reviewing Peer Assess Pro™ activities including previous activities
Automated generation of a unique student-specific Peer Assess Pro™ Survey for all members of the student’s team including their self assessment
Automated email communication to the students of the survey URL required for their completion of the assessment
Warning alerts that provide prompts for actions the teacher can take to resolve the warning. For example, dispatch a proforma email to a student requiring them to reconsider and, optionally resubmit the survey if they were significantly over generous in assessing one team member when compared with other teammates’ assessments of that same team member
Automated email reminders to students to complete the survey despatched 72 hours, 24 hours and 12 hours before the Due Date for the assessment
Mechanisms that reduce the chance of a students making errors in identifying the names of their correct team members. Furthermore, if a teacher makes a mistake, such as forgetting to include a student in a team, there is a recovery mechanism that impacts solely the students in the affected team(s) rather than the class as a whole

Peer Assess Pro™ is a work in progress

We appreciate your participation in this pre-market release of our substantially revised Peer Assess Pro™ in conjunction with the Xorro advanced quiz and survey platform.

As we proceed through this pre-market refinement phase we respond almost daily to your suggestions for improving both the software applications and user documentation. These improvements are implemented at anytime whilst we undergo our Beta Development phase. We anticipate that our implementations are robust enough to prevent loss of your data and wasting your time. We crave your forgiveness if we have been over optimistic in keeping Murphy’s Law at a distance.

Where’s the latest?

You need not take any action to use the latest versions of the Peer Assess Pro™ Xorro Teacher’s Dashboard. Those updates happen in the background and will automatically use any data and activities you have initiated. However, if you use the PDF version of this user guide, you will need to update regularly to the latest version here.

Quick links and related information

FAQ: where may I view the most recent version of the Reference Guide?

FAQ: How do I find the Peer Assess Pro Xorro Teacher’s dashboard?

If you quit your browser then wish to return to the Teachers Dashboard

HOME: Running Activities

Login to your Xorro Teacher’s Dashboard
Navigate to the HOME tab
Note Running Activities should now show your new activity
Select your specific activity, such as: ‘My first trial’
The Peer Assess Pro Teacher’s dashboard will display showing progress for that activity

Alternative method: ACTIVITIES: Running Activities

Login to your Xorro Teacher’s Dashboard
Navigate to the ACTIVITIES tab
Select ‘Running Activities’
Select the specific activity

From HOME Tab

From Activities Tab: Running Activities

Quick links and related information

FAQ: How do I navigate the PARTICIPANTS page for Peer Assess Pro?

Select PARTICIPANTS Tab

Note the list of ‘All participants’ currently known to Xorro in your institution.

Note a list of all other Groups uploaded by other Teachers in your institution. A group is a list of participants, such as students in a class. The minimum requirement for a Group is id, first name, last name.

However, for a peer assessment activity a Group must include team membership for all team members. This team membership is not required for most other Xorro activities. Accordingly, Groups set up for other teachers or by other teachers will rarely contain the correct team membership data required for your Peer Assess Pro™ activity.

Orientation note: Select an existing Group

Select Group ClassAM101.6. This group selection displays a list of about 25 students in the class titled AM101.6

Inactive functions in PARTICIPANTS page

The following buttons have no relevance to creating and launching a Peer Assess Pro activity within Xorro:
Create Participant - REASON: Unable to supply Team name or email FUTURE DEVELOPMENT required
Add to Group - Unpredictable results. Copying a member from one group to another will probably NOT include the required team membership information.
Gradebook - The Gradebook for Peer Assess Pro is generated within the Peer Assess Pro Dashboard
Edit Participant - Unable to supply Team name or email FUTURE DEVELOPMENT required

Quick links and related information

FAQ: How do I correct the participants (team members) in a group I uploaded?

FAQ: How do I correct the Team Composition in a running peer assessment activity?

FAQ: How do I correct the team composition in a running peer assessment activity?

In a launched, running peer assessment activity, you often need need to make these adjustments:

Add a new participant to a team
Delete a participant from a team
Reassign a participant to another team
Correct the id or other details for a participant
Add, delete, or rename a team

Select the context you require

LMS (Moodle, Canvas) Adjust the team composition in a running peer assessment activity on an LMS

Xorro Adjust the team composition in a running peer assessment activity on Xorro

Adjust the team composition in a running peer assessment activity on an LMS

Context: Peer Assess Pro running on an LMS (Moodle, Canvas, Blackboard)

Take care! Here there be dragons!!

Ensure you read ALL of this FAQ before proceeding.

If you make a mistake in this process the consequence may lead to unrecoverable loss of survey responses received to date.

Overview

In the LMS version of Peer Assess Pro, the facilitator is alerted to requests to adjust the team composition in the running activity from several Active Warnings. These Active Warnings and their associated available actions streamline the facilitator’s workflow for managing adjustments to the team arrangement.

For example, new participants may have been enrolled on the LMS since the peer assessment activity was launched. Additionally, a participant may have alerted the facilitator that team participant(s) (or themself) have been

Assigned into an incorrect team
Withdrawn from the course
An inactive contributor to the team’s project.

The relevant Active Warnings are described in these FAQs

FAQ - What is an adjusted team arrangement request? WARNING 0006

FAQ - What is an inactive team member? WARNING 0048

FAQ - How do I resolve an unsynchronised team arrangement? ACTIVE WARNING 0021

The facilitator responds to the suggested actions presented on the Teachers Dashboard for the relevant Active Warnings to first produce a proposed team arrangement on the LMS side. The facilitator will

Confirm or adjust a request by a participant that another participant(s) (or themself) should be reassigned into another existing team, (in response to WARNING 0006 and WARNING 0048). When the facilitator confirms this request Peer Assess Pro automatically reassigns the participant to the correct team on the LMS, but DOES NOT synchronise the reassignment with the running Peer Assess Pro activity
Apply the standard LMS team arrangement facilities to add, remove, and reassign participants to teams (in response to WARNING 0048) and/or
Apply the standard LMS team arrangement facilities to add, remove or rename teams.

The facilitator now must

Review the proposed changes to the arrangement of participants and/or teams on the LMS. Depending on the LMS in use this proposed team arrangement is termed the LMS grouping, group set, or teamset
Examine carefully the Team Composition shown on the Peer Assess Pro Teacher Dashboard Available Actions. The Team Composition explicitly highlights the differences in team membership that exist between the LMS (proposed arrangement) and the running Peer Assess Pro activity. Since there are differences, Active Warning 0021 Unsynchronised team arrangement will be raised on the Teacher Dashboard
FINALLY, the facilitator initiates Synchronise All, the available action in Active Warning 0021 Unsynchronised team arrangement.

What happens with Synchronise All?

Before the facilitator initiates Synchronise All, the refreshed Team Composition view for the Peer Assess Pro activity will now show the updated team arrangement, Step 5 above.The updated view combines the intentions just-stated by the facilitator on the LMS (the proposed team arrangement), and the current status of the team arrangement within the running activity. The symbols (+, -) signal mismatches, or the lack of synchronisation between the LMS and the current Peer Assess Pro activity.

Precautions before Synchronise All!

Prior to the facilitator committing to synchronisation, the facilitator’s proposed changes are clearly indicated, as the example illustrates. Crucially, this Team Composition view provides a ‘last chance’ opportunity for the facilitator to review carefully the changes they intend.

In the Team Composition view, the symbol (-) signifies that, following synchronisation, a participant would be dropped from a team, or a team deleted from the current peer assessment activity. The symbol (+) indicates that a participant or team will be added.

For example, in the following example, once the facilitator has initiated Synchronise All, Kamryn MILLER will be reassigned from team Black Robins to Team Brown Kiwis. The two teams Black Robins and Brown Kiwis will now include the new class participants Jill ROBERTSON and Jason SMITH. The team Wax Eyes will be added, along with its team members, Kael BRIDGES, Jonathan CHANG, and Kyleigh COHEN.

Some survey responses might be deleted!

Note that any survey responses already generated by Kamryn Miller for team Black Robins will be irretrievably deleted following synchronisation. Similarly, Estrella Hawkins’ responses will be irretrievably deleted..

Team composition view prior to synchronise all

Black Robins

Kamryn MILLER (-), Alexander SAMPSON , Mikaela RAY , Ramon MCKNIGHT, Jill ROBERTSON (+)

Brown Kiwis

Estrella HAWKINS (-), August DAUGHERTY , Nehemiah MCCONNELL, Kamryn MILLER (+), Jason SMITH (+)

Grey Warblers

Joslyn HOOVER , Alyvia YANG , Mariyah POLLARD , Arianna SCHROEDER

Pukekos

Dorian SULLIVAN, Elisha NUNEZ , Muhammad HOLT , Skylar MCCLURE

Red Rooks < Red Ruru

Alberto UNDERWOOD , Annika KLINE , June MCKINNEY , Jaylee MURRAY

Waxeyes (+)

Kael BRIDGES (+), Jonathan CHANG (+), Kyleigh COHEN (+)

Team composition following synchronise all

Black Robins

Alexander SAMPSON , Mikaela RAY , Ramon MCKNIGHT, Jill ROBERTSON

Brown Kiwis

August DAUGHERTY , Nehemiah MCCONNELL, Kamryn MILLER, Jason SMITH

Grey Warblers

Joslyn HOOVER , Alyvia YANG , Mariyah POLLARD , Arianna SCHROEDER

Pukekos

Dorian SULLIVAN, Elisha NUNEZ , Muhammad HOLT , Skylar MCCLURE

Red Rooks

Alberto UNDERWOOD , Annika KLINE , June MCKINNEY , Jaylee MURRAY

Waxeyes

Kael BRIDGES, Jonathan CHANG, Kyleigh COHEN

Survey notifications

The founding members of teams Black Robins and Brown Kiwis will be alerted by an automated notifications from Peer Assess Pro they must submit an updated survey response for their re-configured teams, as will the relocated Kamryn MILLER. Notification 0013: RESUBMIT peer assessment due to TEAM CHANGE.

The newly-enrolled class members in the new team Waxeyes Kael BRIDGES, Jonathan CHANG, Kyleigh COHEN, Kael BRIDGES, Jonathan CHANG and Kyleigh COHEN will be advised to submit the peer assessment survey. Notification 0011: Request to COMPLETE peer assessment.

Note that if a founding participant has not yet submitted, they will receive an updated notification to complete the peer assessment, updated to include the new team arrangement, Notification 0011: Request to COMPLETE peer assessment.

There will be no impact on the survey responses for the participants in teams Grey Warblers and Pukekos.

Adding orphan teams to a running activity

An orphan team is a team that has not been assigned to a teamset (grouping, group set).

Teams that are to be added to the activity must be added to the same teamset (grouping, group set) that was used to create and launch the peer assessment activity. Furthermore, ALL the teams available in the teamset must have been selected for the initial launch.

If the Peer Assess Pro activity was launched using orphan teams, then no additional teams can be added. In this case, participants can only be added to or deleted from teams used to create the initial activity.

Why was my Synchronise All action rejected?

A Synchronise All action will rejected for several reasons when the team arrangement is not feasible.

Do not assign a participant to two or more teams within the same teamset, (grouping, group set)
Ensure new teams are allocated to the teamset, (grouping, group set) that was used to launch the peer assessment activity
If the Peer Assess Pro activity was launched using orphan teams, then no additional teams can be added.

Good practice hint when creating a peer assessment activity

Before launching a Peer Assess Pro activity always

Create your teams (groups) on the LMS
Arrange your teams into a teamset (grouping, group set)
Select teamset overview on the LMS to carefully check that

All the participants you require are allocated to a team
No participant is allocated to two or more teams within the same teamset

Most important! Launch the Peer Assess Pro activity using a group set (grouping) AND select ALL the groups in the group set.

Quick links and related information

FAQ - What is an adjusted team arrangement request? WARNING 0006

FAQ - What is an inactive team member? WARNING 0048

FAQ - How do I resolve an unsynchronised team arrangement? ACTIVE WARNING 0021

LMS team arrangement facilities for Moodle

Groups—MoodleDocs. https://docs.moodle.org/400/en/Groups

Grouping users—MoodleDocs. https://docs.moodle.org/400/en/Grouping_users

Groupings—MoodleDocs. https://docs.moodle.org/400/en/Groupings

Groups versus groupings—MoodleDocs. https://docs.moodle.org/19/en/Groups_versus_groupings

Adjust the team composition in a running peer assessment activity on Xorro

Context: Peer Assess Pro running on Xorro

Take care! Here there be dragons!!

Ensure you read ALL of this FAQ before proceeding.

If you make a mistake in this process the consequence may lead to unrecoverable, loss of survey responses received to date.

Key check points

You must upload the entire Participants CSV for all teams and all students you require in your running activity. ‘Entire’ means both adjusted AND the original unchanged data from your original Participants CSV.
Good practice hint: edit a copy of the original Participants CSV you used to create the CSV you used to launch the activity
Make your adjustments to the copy of your original Participants CSV.
Before re-importing, check your new Participants CSV to ensure that you have the correct number of unique teams and unique students that you expect.
Do not upload the corrected Participants CSV to a Xorro group. That will have unpredictable effects.
Re-import the adjusted CSV through the Team Composition selection of the Peer Assessment Running Activity dashboard. Detailed steps below.
Note carefully that the notifications of adjustments match your expectation eg changed team name, email, whatever.

View the team composition

Select the ‘Team Composition’ button for the running Peer Assessment Activity for which you wish to adjust the team composition.

Correct the team composition

Correct the Participants CSV that was used to create the original peer assessment activity. That is, add team members, or swap the team members between teams.
You may correct or add ids and email addresses.
You may add new teams
If you rename teams, then their responses to date will be nulled.
DO NOT adjust the contents of the Column headed ‘group_code’ in your CSV file. It must match exactly the group_code used to launch the original activity. That is, GDE105 in this example.
Review the ‘Key check points’
Re-Import the entire Participants CSV for your class. Not just the changes.

During the re-import, the changes to the teamset will be presented to you so that you can check and confirm the adjustment process. Take care!

Upon completion of the re-import process, the running Peer Assess Pro Activity will continue.

All students in teams affected by a change in composition are now required to resubmit their peer assessment responses. Reason: They now have different team members to rate. The remaining teams of the class will be unaffected. Their responses remain submitted and evident within Peer Assess Pro.

Affected team members will be notified of their need to re-submit by an automatically generated email from Peer Assess Pro.

Subtle technical note

You cannot change the participants in the Xorro Group used to create the running activity, as explained in the FAQ:

FAQ: How do I correct the participants (team members) in a group already uploaded to Xorro?

Reason: whenever a Xorro activity is created a snapshot is taken of the Group used to create the activity. From that moment this snapshot, known as a Xorro Teamset, is inextricably connected with the activity. That activity-specific teamset can be updated only during a running activity through the FAQ detailed above, through the Team Composition section of the Peer Assess Pro dashboard.

In the image above, the Group used to create the peer assessment activity is BT101. Any changes made to that group WILL NOT affect the running activity. The teamset created from the Group BT101 is denoted 2019-02-24 BT101 by Beta Beta. That name indicates what date the teamset was created, from which Group, and by whom.

FAQ: Can I create a peer assessment activity without having all my teams correctly identified by team name and/or team membership?

You can add, swap or delete delete team members anytime before launching the activity, and anytime before the peer assessment activity is finalised.

Good Practice Hint. Get your team composition list absolutely correct before the activity is launched and made available for response by your students. Reason: All students in teams affected by a change in composition will be required to resubmit their peer assessment responses. The students now have different team members to rate. However, the remaining teams of the class will be unaffected.

Quick links and related information

FAQ: How do I correct the Team Composition in a running peer assessment activity?

FAQ: How do I create a CSV file from a Google Sheet?

Good practice hint!

Confirm that the Participants CSV file you created is in the correct format. Open the Participants CSV using a text editor (Apple Textedit, or Microsoft Windows Notepad).

The format should appear as illustrated below. Note that

The first row, containing the column headers must be stated as id, first, last, group_code, team, email. REQUIRED in any order.
The data on each row is separated by the delimiter comma (,). Alternatively, a semicolon (;) may be used. The delimiter is illustrated above in the Sample of participants csv file opened using a text editor.
The definitions and rules for what data may be listed within each of the subsequent rows is explained in Xorro help Importing Participants, Teams and Groups

Sample of participants csv file opened using a text editor

id,first,last,group_code,team,email,

BOWI12,Bob,Wilson,123.101,Tiger,Bob.Wilson@xorroinstitution.com,

ALJO11,Alice,Jones,123.101,Panda,Alice.Jones@xorroinstitution.com,

JOSM13,John,Smith,123.101,Tiger,John.Smith@xorroinstitution.com,

JOSM13,John,Smith,123.202,,John.Smith@xorroinstitution.com,

GRGR15,Greta,Green,123.101,Panda,Greta.Green@xorroinstitution.com,

GRGR15,Greta,Green,123.204,,,

HEJO19,Henry,Jones,123.101,Tiger,Henry.Jones@xorroinstitution.com,

AMTO01,Amanda,Tolley,123.101,Bear,Amanda.Tolley@xorroinstitution.com,

JEWA06,Jeff,Wang,123.101,Panda,Jeff.Wang@xorroinstitution.com,

HOBR03,Holly,Brown,123.101,Bear,Holly.Brown@xorroinstitution.com,

HOBR03,Holly,Brown,123.202,,Holly.Brown@xorroinstitution.com,

THWI18,Thomas,Windsor,123.101,Tiger,Thomas.Windsor@xorroinstitution.com,

ANWO08,Anna,Worth,123.101,Bear,Anna.Worth@xorroinstitution.com,

ANWO08,Anna,Worth,123.202,,Anna.Worth@xorroinstitution.com,

ANWO08,Anna,Worth,123.204,,,

Quick links and related information

FAQ - I’m having problems importing my participants csv

FAQ: How do I view a demonstration version of Peer Assess Pro?

A Beta Test demonstration site has been established with these credentials:

Browse to: https://qf.staging.xorro.com/

Enter: Username BetaTest, Password Secret

This Beta Test User is established for you to view. But don’t touch to hard!

View

A team group list csv file already uploaded to Xorro Participants, titled BT101
A Running Activity for peer assessment using the group BT101 where about half the participant team members have submitted. Note the Teacher’s Dashboard Warnings; the result statistics to date. Please DO NOT FINALISE this activity! Please do not adjust the composition of group BT101 in any csv file you upload
A FINALISED activity where no more responses can be incorporated nor adjustments made to Team Results, Publication Status or Calculation of Personal Result method. (xxx to come xxx)

FAQ: How do I correct the participants (team members) in a group already uploaded to Xorro?

WARNING! HERE THERE BE DRAGONS!!

If a peer assessment activity is launched and running then you cannot update team membership details in that running activity using the procedure described here!!! Instead, apply the procedure described here FAQ: How do I correct the Team Composition in a running peer assessment activity?

Reason: whenever a Xorro activity is created a snapshot is taken of the Group used when creating the activity. From that moment this snapshot, known as a Xorro Teamset, is inextricably connected with the activity. That activity-specific teamset can only be updated during a running activity through the following FAQ.

Update a group’s team members for future use

On the Xorro Dashboard, Select Participants.
In the Groups column left, select the Group name to view a list of all students in your class (Group).

Correct the team members associated with an existing Xorro TeamSet Group

Recreate the Team Members Group CSV file
Load then import the Team Members file according to Section 2. Launch Peer Assessment Activity
Note that the data loaded from the TeamSet CSV operates according to these rules
If the id does not exist in the Xorro Group then ALL new attributes such as id, team, email, first name, last name will ADD a NEW Record to the Group and institution All Participants records.
If the id exists already in the Xorro Group then altered attributes in the CSV, such as team, email, first name, last name will UPDATE the EXISTING record identified by the id in the Group and All Participants records.
Note that team membership is Group specific.
Caution: If the existing Xorro Group contains an id that is NOT supplied in the new csv file then that id will remain in the Xorro group with its original attributes including team membership. HERE THERE BE DRAGONS! You must explicitly delete these surplus members from the Group.

Quick links and related information

FAQ: How do I correct the Team Composition in a running peer assessment activity?

FAQ: Where may I view the most recent version of the user guides?

Quickstart Guide

Quickstart Guide for Peer Assess Pro: Xorro. (2019, March 6). Peer Assess Pro. http://tinyurl.com/pdfQuickWeb

Pdf version: http://tinyurl.com/pdfQuick

Video guides

Launch a Peer Assess Pro Activity. (2019). Auckland: Peer Assess Pro.

Student survey experience. (2019). Auckland: Peer Assess Pro.

Latest reference guide

Peer Assess Pro. (2019, March 5). Manage a Peer Assessment Activity using Xorro: Reference Guide for Teachers. Auckland: Peer Assess Pro

Web version http://tinyurl.com/papRefWeb2

Pdf version http://tinyurl.com/papRefPdf

Work in progress Google DOCS development version

Google Docs version.

Feel welcome to make suggestions or ask questions using the Comment feature of the Google Docs development version. Shows work in progress improvements.

Frequently Asked Questions for teachers and team members

Frequently Asked Questions (FAQs) (2019). In Manage a Peer Assessment Activity using Xorro: Reference Guide for Teachers [web]. Auckland, New Zealand: Peer Assess Pro. http://tinyurl.com/papFAQ

Teachers Process Flowchart

Peer Assess Pro. (2019). Xorro Peer Assess ProTM Teachers Process Flowchart: Overview and Detail. http://tinyurl.com/papChart

Quick links and related information

FAQ: How do I decide which Personal Result method to apply in my peer assessment activity

The choice of calculation method for determining a team member’s personal result is determined by the teacher's preference for compensating more strongly team members who have contributed significantly to their teams, and under-rewarding team members who are peer assessed as weak contributors. The figure illustrates the statistical features, such as team average, range, and standard deviation, associated with each method.

Alternative calculation methods for Personal Result (PR) illustrating effect on team average and spread for a given Team Result

The teacher can select either the Peer Assessed Score (PA Score) or Peer Assessed Index (PA Index) if they wish to exclude a team result in calculating the Personal Result (PR).

More usually, the Peer Assessed Score and Team Result (TR) are combined mathematically to produce a Personal Result. There are three alternative methods. As the figure illustrates, the Indexed Personal Result (IPR) is the least discriminating method, whilst the Rank-Based Personal Result (RPR) is the most discriminating in terms of favouring significant team contributors and penalising weak contributors. Most teachers select the Normalised Personal Result, often with a spread factor of 1.5 to 2.0.

In contrast to the graphical illustration earlier, the following table summarises the example calculations presented through a series of FAQ that present the mathematical definition and example calculations for each method.

Comparison of Personal Results calculated by several methods in a team of four members

	ASSESSEE
ASSESSOR	Bridget	Julian	Lydia	Nigella	Mean	Range
Rank Reversed	1	2	4	3
Peer Assessed Score, PA Score	54	74	82	78	75	28
Peer Assessed Index, PA Index	66	90	100	95	88	34
Team Result, TR	50	50	50	50	50	0
Indexed Personal Result, IPR	33	45	50	48	44	17
Normalised Personal Result, NPR (SpreadFactor = 1)	39	51	56	54	50	17
Normalised Personal Result, NPR (Spreadfactor = 2)	28	52	62	58	50	34
Rank-Based Personal Result, RBR	20	40	80	60	50	60

Source: FAQ: How are peer assessment and personal results calculated and defined mathematically?

Definitions and features of calculation methods used in Peer Assess Pro

Attribute (X1)	Abbreviation (X1)	Definition (X1)
Peer Assessed Score	PA Score	A relative measure of the degree to which a team member has contributed to their team's overall achievement, team processes, and leadership. The Peer Assessed Score (PA Score) is calculated for each team member directly from their Average Team Contribution (ATC) and Average Leadership Contribution (ALC). That is, from the ten components of Team and Leadership contribution survey in the peer assessment. A Peer Assessed score is generally used to compare the relative contribution of students WITHIN the same team, rather than BETWEEN teams. The Team Result has NO impact on the value of the Peer Assessed Score. Values for the PA Score range from zero through 100.
Peer Assessed Index	PA Index	The Peer Assessed Score (PA Score) is indexed upwards so that the person in the team with the highest Peer Assessed Score is awarded a Peer Assessed Index of 100. All other team members receive a proportionally lower PA Index in the ratio PA Score / max(PA Score). The Team Result has NO impact on the value of the Peer Assessed Index.
Team Result	TR	The result awarded to the team for the outputs of their work. The teacher typically derives the Team Result (TR) from grades for team reports, presentations, and results of Team Readiness Assurance Tests. The teacher may select to combine a student's Peer Assessed Index (PA Index) with their team's Team Result (TR) to calculate a Personal Result (PR) for each student, reflecting their relative contribution to the Team Result as assessed by their peer team members. Peer Assess Pro enables the teacher to select from several methods to combine the Team Result and Peer Assessed Index (PA Index) to produce a Personal Result: the Indexed Personal Result (IPR), the Normalised Personal Result (NPR), and the Rank Based Personal Result (RPR).
Measures of a student's personal result
Personal Result	PR	A student's personal result gained from combining their Peer Assessed Index (PA Index) and, optionally, their Team Result (TR). The teacher selects from one of several Calculation Methods to calculate the Personal Result that incorporates the Team Result. These methods are Indexed Personal Result (IPR), Normalised Personal Result (NPR), and Rank-Based Personal Result (RPR). The choice of method is determined by the teacher's preference for compensating more strongly students who have contributed significantly to their teams, and under-reward students who are peer assessed as weak contributors. Figure 1 illustrates the statistical features, such as team average, range, and standard deviation, associated with each method. The IPR is the least discriminating method, whilst the RPR is the most discriminating in terms of favouring significant team contributors and penalising weak contributors, as the figure illustrates.
Indexed Personal Result	IPR	The Indexed Personal Result is calculated from the Team Result (TR) combined with the student's specific Peer Assessed Index (PA Index). The Indexed Personal Result method awards the Team Result to the TOP RATED student in the team, since, by definition, their Peer Assessed Index is 100. All remaining students in the same team earn the Team Result downwards, directly proportional to their PA Index. The Indexed Personal Result calculation means that NO team member can earn an Indexed Personal Result greater than the Team Result. That is, values for the Indexed Personal Result range from zero up to the Team Result.
Normalised Personal Result	NPR	The Normalised Personal Result is calculated from the Team Result combined with the student's specific Indexed Personal Result (IPR). However, in contrast to the IPR method, the Normalised Personal Result method awards the AVERAGE student in the team the Team Result (TR). All remaining students are awarded a Personal Result ABOVE or BELOW the Team Result depending on whether their IPR is above or below that team's average. Features of the Normalised Personal Result are that (a) In contrast to the IPR method, the Normalised Personal Result method calculates a Personal Result ABOVE the Team Result for the above-average peer rated students in the team (b) The average of the team's Normalised Personal Results matches the Team Result (c) The spread of the team's Normalised Personal Results matches the spread of the Indexed Personal Results (IPR) that is calculated for that team. Spread is measured by the standard deviation statistic. . Optional feature: To enhance the effect of rewarding high contributors and penalising weak contributors the tutor can increase the Spread Factor (SF) from the default value of 1.0. Increasing the Spread Factor increases the spread of the results centred around the Team Result. However, an increase in the Spread Factor will maintain a team average NPR that matches that team's Team Result. A Spread Factor of 1.5 to 2.0 is recommended, especially in classes where team members are reluctant to penalise weak contributors and/or reward the highest contributors through their peer assessment rating responses. Values for the NPR range from zero to 100. Calculations that exceed these ranges are clipped to fit within zero to 100
Rank Based Personal Result	RPR	The Rank Based Personal Result is calculated from the Team Result combined with the student's specific Rank Within Team based on that student's Peer Assessed Score. Like the Normalised Personal Personal Result the RPR method awards the AVERAGE student in the team the Team Result. All remaining students are awarded a personal result above or below the Team Result depending on whether their Rank Within Team is above or below that team's middle-ranked student. Features of the Rank Based Personal Result (PR) calculation method are that (a) A team's RPR values are spread over a MUCH WIDER range than the NPR and IPR methods. Small differences in PA scores within a team are amplified significantly by this method (b) In contrast to the IPR method, the RPR method calculates a Personal Result significantly ABOVE the Team Result for the top ranked student in the team (c) Like the NPR method, the average of the team's RPR values matches the Team Result. Values for the Rank Based Personal Result range from zero to 100. Calculations that exceed these ranges are clipped to fit within the range zero to 100.

Note that in the Xorro version of Peer Assess Pro, we have renamed the following Personal Result Methods from those used in the Google Docs version of Peer Assess Pro.

Renaming of terms for Peer Assess Pro

Peer Assess Pro	Abbreviation	Google Peer Assess Pro	Abbreviation
Peer Assessed Score	PA Score	Team Based Learning Score	TBL Score Score
Peer Assessed Index	PA Index	Team Based Learning Index	TBL Index

Quick links and related information

FAQ: How are peer assessment and personal results calculated and defined mathematically?

A teacher has several alternative calculation methods to determine a personal result from a team member’s Peer Assess Pro assessment. The teacher will usually advise team members about the method they have chosen.

The teacher’s choice of calculation method for a personal result is determined by the teacher's preference for

Adjustments to compensate more strongly team members who have contributed significantly to their teams, and under-reward team members who are peer assessed as weak contributors
Incorporating a team result, based on grading of a team’s delivered outputs, such as reports and presentations

These choices are illustrated in this figure.

A student’s Personal Result emerges from the Teacher’s choice of Calculation Method, relative Peer Assessed Score, and Team Result

Calculation methods that exclude a team result

The teacher should select either the Peer Assessed Score (PA Score) or Peer Assessed Index (PA Index) if they wish to exclude the team result in calculating the personal result. Alternatively, set the team result, TR, equal for all teams and use either the IPR, NPR, or RPR methods.

Quick links and related information

FAQ: How is the Peer Assessed (PA) Score calculated?

FAQ: How is the Peer Assessed Index (PA Index) calculated?

Calculation methods that incorporate a team result from team outputs

More usually, the Peer Assessed Score (PA Score) and team result are combined through one of three methods. The following methods are listed in order of increasing impact for compensating more strongly students who have contributed significantly to their teams, and under-rewarding students who are peer assessed as weak contributors

FAQ: How is the Indexed Personal Result (IPR) calculated?

FAQ: How is the Normalised Personal Result (NPR) calculated?

FAQ: How is the Rank Based Personal Result (RPR) calculated?

Quick links and related information

FAQ: What factors are measured in the peer assessment survey?

FAQ: How do I decide which Personal Result method to apply in my peer assessment activity?

FAQ: How do students know where and when to complete the peer assessment activity then review their results?

Automated communications to students

The Teachers Process Flowchart: Detail illustrates the points throughout the peer assessment process where emails are sent to students to advise them

When they can undertake the peer assessment survey
When they can view provisional and finalised results
If they are required to consider resubmitting a response

In most cases, the emails are generated automatically by the Peer Assess Pro system. In the case of warnings, the teacher has the option of initiating an email request to a student, or ignoring that warning.

Copies of all emails are sent to the teacher whose Xorro account was used to launch the activity

Standard operating mode

When you create and launch a Peer Assess Pro™ Peer Assessment activity in Xorro AND the Start Date has been reached:

A unique Participants URL is generated that identifies the assessment survey that your class will complete
The Participants URL is emailed automatically to all students in the class immediately the Start Date for the activity is reached. The email includes details such as: activity title, group_id, teacher name, the due date for completion, and the date when the assessment becomes available for completion. The teacher specified this data when the peer assessment activity was launched and created in the Xorro Teacher’s dashboard
The student must login via the URL supplying the identification code, id specified in the Team List Group of participants associated with the creation of this peer assessment activity
Students undertaking the assessment are sent further emails including the same activity-specific Participants UR when interim and Finalised results are published. Furthermore, they receive additional emails when required. Some emails are initiated by action of the teacher, whilst others are generated automatically. For instance, participants automatically receive a reminder at 72 hours, 24 hours, and 12 hours before the due date if they have NOT completed a submission.
The Peer Assess Pro Log shows a record of all emails sent.

Alternative mode for student access to assessment and results

Alternatively, the teacher can direct students to the Participant URL shown at the top left of the Xorro HOME page. The student must then select from a list the correct peer assessment activity for their response. The teacher may deliver other Xorro-based test activities from which the student must select the correct Peer Assess Pro™ activity distinguished by the Activity Title specified by the teacher.

FAQ: How do I view and experience what the students experience?

FAQ: What questions are asked in the peer assessment survey? in the peer assessment survey?

FAQ: How do I view and experience what the students experience?

View your student’s personal results feedback report directly from your Teacher’s Dashboard

Click on the name of a student, and you will view the feedback report available for the student.

There are four possible views.

Student view. Published. The view the student can view right now.
Student view. Live. The view not yet visible to the student until the teacher Publishes or Updates the Results Publication Status. This view lets the teacher explore student results before committing them to view by the students in the class.
The two Teacher views show results in greater detail. Namely, the ratings, and which students said what about each of their teammates. Again, the teacher can view the results already available for students’ view (Published), or works-in-progress (Live).

The student’s views are anonymised.

View your students’ experience of the Peer Assess Pro™ survey

Enter your Participants’ URL into your browser
Select the activity you wish to experience
Login in using the Identification (id) of a student in the Team List Group you used to create the activity
You will see a variety of screens, depending on the state of the peer assessment survey

Enter your Participants’ URL into your browser

Select the activity you wish to experience

Login in using the Identification (id) of a student in the Team List Group used to create the activity

View a survey ready and waiting for responses

The student will see this view when all of the following are TRUE:

The Activity Start Date has passed
The Activity has NOT been Finalised by the Teacher
The Student has NOT submitted their responses to the Survey.

Note that students can continue to submit responses AFTER the Due Date UNTIL the teacher has Finalised the activity.

View a sample question

View a student’s published results

The student will be able to see their Personal Results when all the following are true:

The Activity Start Date has passed
The date is within two weeks of the activity being Finalised by the Teacher.
The Student has submitted their responses to the Survey.
The teacher has Published results updated to include the student’s response
At least one half of the team members have submitted a response, with at least three submissions required.

A student with a Xorro Plus account may view his results any time after the Activity is Finalised by the Teacher.

The student views

Their Personal Result, Team Result and Peer Assessed Score
A spider chart showing their self assessment compared with the class average, and the ratings by others in their team
Qualitative remarks about Task and Leadership Contribution, and areas for future development

Example results for a student

View the peer assessment survey for a demonstration class

Xxx TO DO xxx

Quick links and related information

FAQ: What questions are asked in the peer assessment survey? in the peer assessment survey?

FAQ: How is the Peer Assessed (PA) Score calculated?

FAQ: How do students know where and when to complete the peer assessment activity then review their results?

FAQ: Why are different terms used to display peer assessment results in the Xorro and previous Google versions of Peer Assess Pro™?

The following terms have been renamed from the Google version of Peer Assess Pro for Peer Assess Pro

Renaming of terms for Peer Assess Pro

Peer Assess Pro	Abbreviation	Google Peer Assess Pro	Abbreviation
Peer Assessed Score	PA Score	Team Based Learning Score	PA Score Score
Peer Assessed Index	PA Index	Team Based Learning Index	PA Index

Quick links and related information

FAQ: How do I decide which Personal Result method to apply in my peer assessment activity?

FAQ: How do I take action on the Warnings presented in the Peer Assess Pro™ Teacher’s Dashboard?’ What if I ignore the Warnings?

In general, see the Sections

Action responses to warnings

Responding to Active Warnings

Warning messages are under constant development and refinement as we respond to facilitators’ and team members’ experience of Peer Assess Pro.

Critical and catastrophic warnings!

These warnings must be resolved, otherwise utterly invalid results will arise, and students’ time will be wasted completing incorrect surveys.

Example: The composition of a team needs adjusting, see

See Adjusting team composition

Important warnings

Peer Assess Pro will not be able to present results for all teams unless these warnings are resolved.

Example: Insufficient responses from a team are received

See Results hidden from team members and teacher

Example: Enter Team Results:

See Enter Team Results

Informational warnings

Advisory warnings do not affect critically the operation of Peer Assess Pro. However, the teacher would be prudent to review the details to ensure that peer assessments have been conducted fairly and honestly.

Example: Overgenerous or parsimonious ratings by a team member.

FAQ: How is an outlier peer assessment rating identified? WARNING 0042

Optional emails generated for team members

Several warnings give the facilitator the option to despatch an email to students advising them of exceptional conditions and requesting their action. For example

Mismatched self-assessment: A team member's self-assessment is materially different from the peer assessment given by their team
Outlier individual rating: A team member has assessed another team member very differently than the other team members.
At risk team member: A team member has been rated amongst the lowest in class
Low quality team rating: A team may have engaged unconstructively with peer assessment

The criteria used to generate these warnings, and the recommended response by the facilitator is detailed in this section:

Responding to Active Warnings

For example, in the case of a Mismatched self-assessment, the team member is invited to meet with the teacher to explore the reasons for the mismatch, and develop approaches to narrow the gap.

Quick links and related information

FAQ: What is the content of emails sent by Peer Assess Pro?

Responding to Active Warnings

FAQ: When, why, and how do I Refresh and Update Results?

When to recalculate

You select Refresh and Update results when

You have adjusted the values for one or more Team Results
You have adjusted the method of calculation for the Personal Result
You are about to Publish results to students
You are about to Download statistics
You are about to Finalise the peer assessment activity
New survey responses have been received by Xorro, but they have not yet been transferred into the Peer Assess Pro database. If in doubt, Refresh!

Why recalculate?

The most important reason is that you as a teacher MUST be able to review results BEFORE displaying (publishing) results to students. After examining the results to date, you might publish an interim snapshot of the results for view by students.

Students may review the interim results and raise issues such as a questionable peer assessment rating, such as scapegoating. Alternatively, you may need to adjust a Team Result, or experiment with another method of Personal Result Calculation.

In this situation, we have presumed you do not want new responses, nor adjustments to be immediately viewable by students. In particular, you need the opportunity to review the effect of adjustments before explicitly publishing revised results to students.

How to recalculate

Select Refresh
Optionally adjust Team Results
Select Peer Assess Pro in the Xorro Teachers Dashboard
Select Recalculate
Optionally, select an alternative Personal Result method of calculation
Confirm Recalculate

Quick links and related information

FAQ: What questions are asked in the peer assessment survey?

The Peer Assess Pro survey measures one overall assessment, Recommendation, followed by ten quantitative ratings, then several qualitative questions.

The ten quantitative ratings are used to calculate the Peer Assessment Score (PA Score). The ten ratings are categorized into two classes: Contribution to Task, and Contribution to Leadership and Teamwork, as shown in the example survey below.

In addition, two qualitative questions are asked that request examples of behaviours supporting the quantitative ratings in relation to Contribution to Task, and Contribution to Leadership and Teamwork. Finally, the assessor is asked to provide Development Feedback. That is, advice that would help the team member improve their future contribution to the team.

Quick links and related information

FAQ - What is the benefit of a standardized peer assessment rubric?

FAQ: How is the Peer Assessed (PA) Score calculated?

FAQ: How do students know where and when to complete the peer assessment activity then review their results?

FAQ: How do I decide which Personal Result method to apply in my peer assessment activity?

The ten questions used as the basis for calculating the Peer Assessment Score are adapted from:

Deacon Carr, S., Herman, E. D., Keldsen, S. Z., Miller, J. G., & Wakefield, P. A. (2005). Peer feedback. In The Team Learning Assistant Workbook. New York: McGraw Hill Irwin.

Example Peer Assessment Survey: Quantitative

My name is:		I am rating my team member:
My Team name is:		Team Member A
			Team Member B
				Team Member C
					Self
Recommendation	How likely is it that you would recommend this team member to a friend, colleague or employee? 1 = Highly unlikely, 5 = Extremely likely
Contribution to Task Accomplishment
	Rate the team member on a 5-point scale. Rating scale: 1 = Almost never, 2 = Seldom, 3 = Average, 4 = Better than most, 5 = Outstanding Rate your typical or average team member a mid-level rating of 3.
Initiative	Shows initiative by doing research and analysis. Takes on relevant tasks with little prompting or suggestion.
Attendance	Prepares for, and attends scheduled team and class meetings.
Contribution	Makes positive contributions to meetings. Helps the team achieve its objectives.
Professionalism	Reliably fulfils assigned tasks. Work is of professional quality.
Ideas and learning	Contributes ideas to the team's analysis. Helps my learning of course and team project concepts.
Contribution to Leadership and Team Processes
Focus and task allocation	Keeps team focused on priorities. Facilitates goal setting, problem solving, and task allocation to team members.
Encourages contribution	Supports, coaches, or encourages all team members to contribute productively.
Listens and welcomes	Listens carefully and welcomes the contributions of others.
Conflict management and harmony	Manages conflict effectively. Helps the team work in a harmonious manner.
Chairmanship	Demonstrates effective leadership for the team. Chairs meetings productively.

Example Peer Assessment Survey: Qualitative

Peer Assessment Survey:

Feedback to the team member

Submit one copy of this form for each team member

My name is:

I am a member of Team Number and Name:

I am assessing (student’s name):

Contribution to Task Accomplishment

For the team member you have assessed, provide specific examples of productive or ineffective behaviours related to your ratings of Contribution to Task Accomplishment. For example, shows initiative; attends meetings; makes positive contributions; helps team achieve objectives; is reliable; contributes quality work; contributes to learning of course concepts. Further examples here http://tinyurl.com/BARSOhland

Contribution to Leadership and Team Processes

For the team member you have assessed, provide specific examples of productive or ineffective behaviours related to your ratings of Contribution to Leadership and Team Processes. For example: keeps team focused on priorities; supports, coaches and encourages team members; listens carefully; manages conflict effectively; demonstrates effective leadership.

Development feedback

What specific behaviours or attitudes would help your team member contribute more effectively towards your team's accomplishments, leadership, and processes? Please provide specific positive or constructive feedback that could enable the team member to improve their behaviour productively. Considering your team member's strengths, how could that person coach other team members to acquire similar strengths for Task Accomplishment, Team Processes, and Leadership?

Source: Peer Assess Pro (2019).

FAQ: How is the Peer Assessed (PA) Score calculated?

The Peer Assessed Score, PA Score, is a relative measure of the degree to which a team member has contributed to their team's overall achievement, team processes, and leadership.

A Peer Assessed Score is generally used to compare the relative contribution of students WITHIN the same team, rather than BETWEEN teams. The Team Result has NO impact on the value of the Peer Assessed Score.

The PA Score is calculated for each team member directly from summing the ten ratings of Team and Leadership Contribution surveyed in the peer assessment. The sum of ratings is adjusted by scale factors to give values for the PA Score that range from zero through 100.

The Peer Assessed Score is an essential factor used as the basis for calculating several alternative measures of Personal Result including the Peer Assessed Index (PA Index), Indexed Personal Result (IPR), Normalised Personal Result (NPR), and Rank Based Personal Result (RPR).

The self-assessment is excluded from calculating PA Score

The self-assessment conducted by a team-member is EXCLUDED from the calculation of their Peer Assessed Score. The self-assessment, PA (self), is used to enable the student to compare their self-perception with that of their team members, and the class as a whole. One method of comparison, the IRSA, is based on the ratio as detailed in the FAQ:

FAQ: How is the Index of Realistic Self Assessment (IRSA) calculated?

Mathematical definition of Peer Assessed Score, PA Score

There are ten Peer Rating components awarded by each Assessor, a, to each Assessee, s, in the team of t members. The mathematical task is to combine all these ratings into one Peer Assessed Score for each team member.

The Peer Assessed SubScore is defined as the peer assessment score awarded by Assessor a to Assessee s:

Where

= the Peer Rating for each of the ten peer assessment components, r, submitted by the Assessor a for the assessed team member, the Assessee, s. The student’s self-assessment is excluded from the calculation of the PA Score. The Recommendation rating is excluded from calculation of the PA Score.

To ensures the PA Score ranges from zero through 100 the following features are required in the above formula:

Since each rating is on a scale of 1 through 5, then 1 must be deducted from each rating to give a scale of zero through 4.
The scale factor (100/40 = 2.5) achieves the required range of zero through 100 for the calculated PA Score.

The Peer Assessed Score, for team members s is the mean of the PA Subscores awarded by the other (t - 1) team members to the team member s.

Where

t = the number of team members in the team in which s is a team member.

= the peer assessment score awarded by Assessor a to Assessee s, mathematically defined earlier.

Note that Peer Assessed Score takes NO account of the team’s Team Result. The Team Result is accounted for in the Indexed Personal Result (IPR), Normalised Personal Result (NPR) and Rank-Based Personal Result (RPR) methods discussed elsewhere.

An example calculation is shown below. In the first table, the team member Bridget (ASSESSEE) is rated by her three team members (ASSESSORS), plus her own self-rating. The subsequent tables show the calculation of the Peer Assessment Score for all four team members based on all team members’ assessment ratings. The long-form calculations show in detail the arithmetic calculations.

Quick links and related information

FAQ: What questions are asked in the peer assessment survey?

FAQ: How do students know where and when to complete the peer assessment activity then review their results?

Alternative but equivalent methods for calculating the Peer Assessed Score are detailed below in the section:

Alternative mathematical formulations of PA Score

Example table of assessments for assessed team member Bridget

ASSESSEE: Bridget		ASSESSOR: Ratings by team member:
Team Name: Kubla		Bridget (Self)
			Julian
				Lydia
					Nigella
						Mean Rating
Contribution to Task Accomplishment
	Rating scale: 1 = Almost never, 2 = Seldom, 3 = Average, 4 = Better than most, 5 = Outstanding
Initiative	Shows initiative by doing research and analysis. Takes on relevant tasks with little prompting or suggestion.	5	5	3	1	9/3
Attendance	Prepares for, and attends scheduled team and class meetings.	4	4	4	1	9/3
Contribution	Makes positive contributions to meetings. Helps the team achieve its objectives.	4	5	5	1	11/3
Professionalism	Reliably fulfils assigned tasks. Work is of professional quality.	4	3	4	1	8/3
Ideas and learning	Contributes ideas to the team's analysis. Helps my learning of course and team project concepts.	5	5	5	1	11/3
Contribution to Leadership and Team Processes
Focus and task allocation	Keeps team focused on priorities. Facilitates goal setting, problem solving, and task allocation to team members.	5	5	3	1	9/3
Encourages contribution	Supports, coaches, or encourages all team members to contribute productively.	4	4	4	1	9/3
Listens and welcomes	Listens carefully and welcomes the contributions of others.	5	5	3	1	9/3
Conflict management and harmony	Manages conflict effectively. Helps the team work in a harmonious manner.	4	4	4	1	9/3
Chairmanship	Demonstrates effective leadership for the team. Chairs meetings productively.	5	5	5	1	11/3
SubTotal	SubTotal = Task + Leadership	45	45	40	10	# 95/30 ( 3.167)
Peer Assessed Score	PA Score = (2.5 x SubTotal ) - 25	* 87.5	87.5	75	0	54.2
* The self-assessment ratings are excluded from calculation of the PA Score. So, 54.2 = (87.5 + 75 + 0) / 3 # Alternatively, PA Score = (25 x Mean Rating) - 25. So, 54.2 = 25 x 95/30 - 25 = (25 x 3.167) - 25

Example calculations of Peer Assessed Score

Suppose that the Peer Assessed Scores determined from all four team members rating each other appear as follows. Bridget’s PA Scores are copied from the previous table, forming the second vertical column here.

Since

Now consider the Assessment by Lydia of Bridget

In the previous table, note how Nigella rated Bridget with the minimum possible rating of one for all ten components. By definition, that gives a PA Score of zero. Similarly, if an assessor had rated a team member the maximum rating of five across all ten components, then a PA Score of 100 would have resulted.

Peer Assessed Sub-Scores for a team of four members

	ASSESSEE
ASSESSOR	Bridget	Julian	Lydia	Nigella
Bridget	87.5	62.5	75	72.5
Julian	87.5	92.5	87.5	82.5
Lydia	75	82.5	77.5	80
Nigella	0	77.5	82.5	82.5

Now the PA Score for each ASSESSEE team member is calculated from the mean of the PA SubScores provided by the other ASSESSORS in their team, as shown in the following table. The self-assessments of each ASSESSOR are excluded from the calculation. For example, the PA Score for Nigella is determined as follows from the ratings by her three teammates Bridget, Julian and Lydia:

Since

Then for Nigella

Calculation of Peer Assessed (PA) Scores for a team of four members

	ASSESSEE
ASSESSOR	Bridget	Julian	Lydia	Nigella
Bridget	-	62.5	75	72.5
Julian	87.5	-	87.5	82.5
Lydia	75	82.5	-	80
Nigella	0	77.5	82.5	-
Peer Assessed Score	54.2	74.2	81.7	78.3

Note how Nigella’s rating of Bridget (PA Score = 0) seems an outlier when compared with the much higher ratings given by Julian and Lydia (7.5 and 75). Peer Assess Pro warns the teacher when outlier ratings like this occur.

This outlier issue is discussed in

FAQ: How is an outlier peer assessment rating identified? WARNING 0042

Alternative mathematical formulations of PA Score

The following equations provide the identical mathematical result for the calculation of PA Score.

Calculation from Average Rating

Where:

Average Rating is the average rating of an assessed student s averaged over all the ten components of the rating for that student, by their team members. The Average Rating lies between 1 and 5.

The factor (-1) adjusts the Average Rating value to zero through 4. The scale factor 100 /4 adjusts the PA Score to lie between zero and 100.

Notice from the first table showing ratings of Bridget that the average rating across all ten components contributing to her Peer Assessment Score given by her three team members was shown as

Therefore, the PA Score is calculated directly from the average rating:

Calculation from Average Team and Leadership Contributions

Finally,

Where:

ATC and ALC are the average ratings for the five components that comprise the Task and Leadership contributions, respectively.

Mathematically:

ATC and ALC range over the values 1 through 5. The factor (-1) adjusts those values from zero through 4. The scale factor 50/4 (= 12.5) ensures that the PA Score achieves a range from zero to 100.

Quick links and related information

FAQ: How do I decide which Personal Result method to apply in my peer assessment activity?

FAQ: How are peer assessment and personal results calculated and defined mathematically?

FAQ: How is the self-assessment used to calculate Peer Assessed Score?

The self-assessment conducted by a team-member when they rate their team members is EXCLUDED from the calculation of that team member’s Peer Assessed Score. Instead, their self-assessment, PA (self), is used to enable the team member to compare their self-perception with that of their team members, and the class as a whole. This comparison is provided to the team member through a SPider Chart and the calculation of their Index of Realistic Self Assessment (IRSA).

Spider chart of individual and averaged team peer ratings

The Spider Chart shows each of their eleven ratings provided by themself, compared with the average of the ratings provided to them by their peer team members. The class average ratings for each of the 11 factors are also provided. In this example, the team member has significantly UNDERRATED themself on nearly all factors (innermost plots), when compared with the ratings provided by their team members (orange).

Spider Chart comparison of self and other team members’ contribution ratings

Index of Realistic Self-Assessment (IRSA)

Another method of comparison, the IRSA, is based on the ratio

as detailed in the FAQ:

FAQ: How is the Index of Realistic Self Assessment (IRSA) calculated?

For the team member illustrated in the foregoing Spider Chart, their Peer Assessed Score, PA Score, is 92 and their self-assessed Score, PA (self), is 75. The ratio results in the Index of Realistic Self Assessment (IRSA) 122 = 100 x 92 / 75.

An IRSA between 75 and 95 is typical of about 2/3 of team members in a class. About ⅙ of team members achieve an IRSA below 75. Such people appear to assess their team members excessively OVERCONFIDENT in their abilities. In contrast, an IRSA above 95 suggests the team member has a tendency to UNDERESTIMATE their team contribution when contrasted with the assessment perceived by their team members.

Quick links and related information

FAQ: How are peer assessment and personal results calculated and defined mathematically?

FAQ: How is the Peer Assessed Index (PA Index) calculated?

The Peer Assessed Index is defined such that the team member with the maximum PA Score for each team is assigned a PA Index of 100. All other team members in the same team are scaled in relation to the maximum PA Score for that group.

In a gradebook of results, the PA Index is useful for identifying the team members most highly rated by their peers, as they have PA Indexes in the 90 to 100 range. In combination with the Team Result, the PA Index is used to calculate the Indexed Personal Result, (IPR), Normalised Personal Result, (NPR) and Rank-Based Personal Result (RPR).

Mathematical definition of Peer Assessed Index

Where

= the Peer Assessed Score for a team member s in team t, as defined in: FAQ: How is the Peer Assessed (PA) Score calculated?

= the maximum value of PA Score found across all members in team t.

Example calculations of Peer Assessed Index

Consider a team of four team members, whose PA Scores are shown in the following table. Lydia has a PA Score of 82, the highest for the team. Therefore, Lydia’s PA Index is 100, by definition.

Calculation of Peer Assessed Index (PA Index) for a team of four members

	ASSESSEE
ASSESSOR	Bridget	Julian	Lydia	Nigella
Bridget	-	62.5	75	72.5
Julian	87.5	-	87.5	82.5
Lydia	75	82.5	-	80
Nigella	0	77.5	82.5	-
Peer Assessed Score	54	74	82	78
Peer Assessed Index	66	90	100	95

Bridget has a PA Score of 54, the lowest for the team. Therefore, since

Note that, as expected

The data for the previous table is drawn from

FAQ: How is the Peer Assessed (PA) Score calculated?

Quick links and related information

FAQ: How do I decide which Personal Result method to apply in my peer assessment activity?

FAQ: How is the Peer Assessed (PA) Score calculated?

FAQ: How is the Indexed Personal Result (IPR) calculated?

The Indexed Personal Result (IPR) is calculated from the Team Result (TR) combined with the team member’s specific Peer Assessed Index (PA Index). The Indexed Personal Result method awards the Team Result to the TOP RATED team member in the team, since, by definition, their Peer Assessed Index is 100. All remaining team members in the same team earn the Team Result downwards, directly proportional to their PA Index.

The definition of Indexed Personal Result means that NO team member can earn an Indexed Personal Result greater than the Team Result. That is, values for the Indexed Personal Result range from zero up to the Team Result. Consequently, the IPR disadvantages team members who have been rated unfavourably by their peers. However, no reward is made for the team member(s) who have been rated as the most contributing team members. In contrast, the Normalised Personal Result and Rank-Based Personal Result do award a Personal Result above the Team Result for those team members who contribute above average to the team’s outputs, as assessed by their peers.

Mathematical definition of Indexed Personal Result

For each team member s, in their team, t

Where

= the team result awarded by the teacher for the outputs of team t

= the Peer Assessed Index for the team member s, as defined in

FAQ: How is the Peer Assessed Index (PA Index) calculated?

Example calculations of Indexed Personal Result

Suppose that the following team has a Team Result, TR, of 50 and Peer Assessed Indexes previously calculated as follows. The example data is taken from:

FAQ: How is the Peer Assessed Index (PA Index) calculated?

Calculation of Indexed Personal Result in a team of four members

	ASSESSEE
ASSESSOR	Bridget	Julian	Lydia	Nigella
Peer Assessed Score, PA Score	54	74	82	78
Peer Assessed Index, PA Index	66	90	100	95
Team Result, TR	50	50	50	50
Indexed Personal Result, IPR	33	45	50	47.5

Bridget has a PA Index of 66, the lowest for the team. Therefore, since

In contrast, Lydia has the highest PA Score in the team, and hence a PA Index of 100. Therefore

The IPR for Lydia is equivalent to the Team Result, 50, as defined.

Quick links and related information

FAQ: How do I decide which Personal Result method to apply in my peer assessment activity?

FAQ: How is the Peer Assessed Index (PA Index) calculated?

FAQ: How is the Peer Assessed (PA) Score calculated?

FAQ: How is the Normalised Personal Result (NPR) calculated?

The Normalised Personal Result, NPR, is calculated from the Team Result combined with the team member’s specific Indexed Personal Result (IPR). The Normalised Personal Result method awards the average student in the team the Team Result (TR). All remaining students are awarded a Personal Result above or below the Team Result depending on whether their IPR is above or below that team's average IPR.

Features of the Normalised Personal Result method are that

In contrast to the IPR method, the Normalised Personal Result method calculates a Personal Result above the Team Result for the team members peer rated above-average in the team
The average of the team's Normalised Personal Results will match that team’s Team Result
The spread of the team's Normalised Personal Results matches the spread of the Indexed Personal Results (IPR) that is calculated for that team. Spread is measured by the standard deviation statistic.
The teacher can choose to s t r e t c h the spread, as explained later, using a spread factor. A higher spread factor increases the rewards to the more highly peer assessed team members in the team, whilst conversely applying a greater penalty to poorly assessed team members.

Use the Normalised Personal Result method with a high Spread Factor if you

Consider your class of students tends to be cautious in peer assessing positively the team members who contribute especially well to their teams
Consider your class of students tend to be cautious in peer assessing negatively the team members who contribute especially poorly to their teams
Want to signal strongly that significantly above average contribution to team work is acknowledged and rewarded by the teacher
Want to signal strongly that significantly below average contribution to team work is noticed and penalised by the teacher.

Mathematical definition of Normalised Personal Result

For each team member s, in their team, t

Where

the team result awarded by the teacher for the outputs of team t

That is, the mean value of the IPR values found for team t, containing n team members.

= a factor chosen optionally by the teacher that will S T R E T C H each team’s intrinsic spread of NPRs, as measured by the team’s standard deviation of NPR results. The default Spread Factor is 1.0. However a Spread Factor of between 1.5 and 2.o is recommended.

Values of NPR are trimmed to be within the range zero to 100.

Example calculations of Normalised Personal Result

Suppose that the following team has a Team Result, TR, of 50 and Indexed Personal Result previously calculated as follows. This first example illustrates a Spread Factor of 2.0. The example data is taken from:

FAQ: How is the Indexed Personal Result (IPR) calculated?

Calculation of Normalised Personal Result in a team of four members

Spreadfactor = 2.0

	ASSESSEE
ASSESSOR	Bridget	Julian	Lydia	Nigella	Mean
Peer Assessed Score, PA Score	54	74	82	78
Peer Assessed Index, PA Index	66	90	100	95
Team Result, TR	50	50	50	50
Indexed Personal Result, IPR	33	45	50	48	44
Correction Factor (Spreadfactor = 2)	-22	+2	+12	+8	0
Normalised Personal Result, NPR (Spreadfactor = 2)	28	52	62	58	50

Bridget has a PA Index of 66, the lowest for the team.

The for the four-member team is 44, calculated from ¼ x (33 + 45 + 50 + 48).

Since

Then

In contrast, the Normalised Personal Result for Lydia, with her IPR of 50, is calculated as follows:

Note how Lydia’s NPR of 62 is above the team Result of 50. Note also how the mean of the NPR values across the team is 50 = (28 + 52 + 62 + 58)/4, identical to the Team Result of 50.

Impact of adjusting the Spread Factor on Normalised Personal result

The previous example showed calculations of NPR using a Spread Factor of 2.0. The following table shows the results of calculating the Normalised Personal Result for the team using a more modest Spread Factor of 1.0.

Note the following:

Bridget achieves an NPR of 39 using a Spread Factor of 1, compared with her achieving the lower NPR of 28 when using a Spread Factor of 2
In contrast, Lydia’s NPR is raised from 56 to 62 when using a Spread Factor of 2
The higher Spread Factor of 2 increased the range of NPR values in the team from 17 (= 56 - 39) to 34 (= 62 - 28), a doubling of spread.
The mean value of the NPRs earned by the team remains the same under both Spread Factors, and is exactly the same as the Team Result, 50.

The default Spread Factor is 1.0. However a Spread Factor of between 1.5 and 2.o is recommended.

Calculation of Normalised Personal Result in a team of four members

SpreadFactor = 1.0

	ASSESSEE
ASSESSOR	Bridget	Julian	Lydia	Nigella	Mean
Peer Assessed Score, PA Score	54	74	82	78
Peer Assessed Index, PA Index	66	90	100	95
Team Result, TR	50	50	50	50	50
Indexed Personal Result, IPR	33	45	50	48	44
Correction Factor (SpreadFactor = 1)	-11	+1	+6	+4	0
Normalised Personal Result, NPR (SpreadFactor = 1)	39	51	56	54	50

Quick links and related information

FAQ: How do I decide which Personal Result method to apply in my peer assessment activity?

FAQ: How is the Peer Assessed Index (PA Index) calculated?

FAQ: How is the Peer Assessed (PA) Score calculated?

FAQ: How are peer assessment and personal results calculated and defined mathematically?

FAQ: How is the Rank Based Personal Result (RPR) calculated?

The Rank Based Personal Result is calculated from the Team Result combined with the student's specific Rank Within Team based on that student's Peer Assessed Score. Like the Normalised Personal Personal Result the RPR method awards the AVERAGE student in the team the Team Result. All remaining students are awarded a personal result above or below the Team Result depending on whether their Rank Within Team is above or below that team's middle-ranked student.

The Extended version of the RPR method, due for implementation in 2022, incorporates the use of a Spread Factor.

Features of the Rank Based Personal Result (RPR) calculation method are that

Conceptually, the team results are imagined as the key ingredient for a cake related in size to the number of team members. The cake is next sliced into equal sized pieces of cake. Each member of the team is awarded one or more pieces of cake. The most poorly rated team member, measured by Peer Assessed Score, receives one piece of cake. The next best team member receives two pieces of cake, and so forth.
The number of pieces of cake to be shared equals the sum of the rankings, which equals . For example, for a team of four members, there will be ten pieces of cake to be shared.
Half shares of cake are allocated in the case of tied rankings amongst an even number of students. Otherwise, every team member gets an integer number of pieces.
A team's RPR values range much wider than the NPR and IPR methods, when a Spread Factor of 2.0 is deployed. Small differences in PA scores within a team are amplified materially by this method.
In contrast to the Indexed Personal result (IPR) method, the RPR method calculates a Personal Result significantly ABOVE the Team Result for the top ranked student in the team
Like the Normalised Personal Result (NPR) method, the mean of the team's RPR values matches the Team Result.
Values for the Rank Based Personal Result range from zero to 100. Calculations that exceed these ranges are clipped to fit within the range zero to 100.

Mathematical definition of Rank-Based Personal Result

For student s in their team t with n team members

Where

the team result awarded by the teacher for the outputs of team t

The number of pieces of cake allocated to team member s. An integer number, except in the case of tied ranks.

= the rank average of the team member s in team t containing n members, where the team member with the highest Peer Assessed Score in that team is ranked as 1. Equal ranks are permitted, and calculated using the rank.average method.

= numbers of members in team t

Values of RPR are trimmed to lie within the range zero to 100.

Example calculations of Rank-Based Personal Result

Suppose that the following team with n = 4 team members has a Team Result, TR, of 50 and Peer Assessed Scores previously calculated. The example data is taken from:

FAQ: How is the Peer Assessed (PA) Score calculated?

Calculation of Rank-Based Personal Result in a team of four members

	ASSESSEE
ASSESSOR	Bridget	Julian	Lydia	Nigella	Mean
Peer Assessed Score, PA Score	54	74	82	78
Rank Average	4	3	1	2
Share Fraction	1/10	2/10	4/10	3/10
Team Result, TR	50	50	50	50	50
Rank-Based Personal Result, RBR	20	40	80	60	50

Observe how there are ten pieces of cake to be allocated, calculated from the sum of the n=4 ranks.

Thus, the poorest ranking team member, Bridget, gets one piece, a 1/10 share fraction. In contrast, the best ranked team member, Lydia, gains four pieces, a 4/10 share fraction.

Bridget has a PA Score of 64, the lowest for the team. Her rank in the team, is therefore 4.

Note how Julian receives double the ShareFraction, and, consequently, double the RPR than does Bridget

Lydia, the top-ranked student, = 4, receives four times the RBR that Lydia received, = 80.

Observe how the mean of the RBR values matches the Team Result for team t of 50.

Observe that, by definition, the sum of the ShareFractions across the team is exactly 10/10 = 100 %. All the ten pieces of cake are allocated across the team, and each person gets one more piece than the next poorer ranked student (except in the case of ties)

Example calculation with tied ranks

The following example shows a case where two team members, Julian and Nigella, have the same Peer Assessed Score of 74, and a rank of 2.5 = (2+3)/2. The Google RANK.AVG function where the is_ascending flag is set to 1 demonstrates this ranking behaviour.

Calculation of Rank-Based Personal Result with tied scores

	ASSESSEE
ASSESSOR	Bridget	Julian	Lydia	Nigella	Mean
Peer Assessed Score, PA Score	54	74 =	82	74 =
Rank Average	4	2.5 =	1	2.5 =
Share Fraction	1/10	2.5/10	4/10	2.5/10
Team Result, TR	50	50	50	50	50
Rank-Based Personal Result, RBR	20	50	80	50	50

Alternative mathematical formulation

The formula defined above can be simplified for calculation purposes by substituting the formula for ShareFraction.

Adjusting the range using a spread factor

In a manner similar to the use of the Scale Factor for the Normalised Personal Result Method, NPR, a Spread Factor can be applied in calculation of the Rank-Based Personal Result.

A SpreadFactor of 2.0 provides the same ‘natural’ values of RBR as determined by the earlier formula. The default Spread Factor is 1.0, which reduces the ‘natural’ spread given by the earlier definition, and brings the spread of RBR values to match the values typically found using the Normalised Personal Result Method with the default SpreadFactor of 1. A Spread Factor of between 1.5 and 2.o is recommended. The following example shows the calculations for RBR with Spread Factors of 2.0, 1.0, and 2.5.

Calculation of Rank-Based Personal Result with several spreadfactors

	ASSESSEE
ASSESSOR	Bridget	Julian	Lydia	Nigella	Mean	Range
Peer Assessed Score, PA Score	54	74 =	82	74 =		28
Rank Average	4	2.5 =	1	2.5 =
Share Fraction	1/10	2.5/10

Everyone	For teachers	For team members
Xorro-Q help	Login to Xorro-Q	Join peer assessment activity
www.peerassesspro.com	The peer assessment survey	The purpose of peer assessment
Reference guide	Login and orientation	Undertake the peer assessment
FAQs on the web at http://tinyurl.com/papFAQ	Launch peer assessment activity	Use peer assessment results for better performance
Videos	Manage the peer assessment activity
Quickstart guide for teachers	Definitions, calculations, and examples
Contact us	eBook - 7step guide

Everyone	For teachers	For team members
Xorro-Q help	Login to Xorro-Q	Join peer assessment activity
www.peerassesspro.com	The peer assessment survey	The purpose of peer assessment
Table of contents: Reference guide	Login and orientation	Undertake the peer assessment
FAQs on the web at http://tinyurl.com/papFAQ	Launch peer assessment activity	Use peer assessment results for better performance
Videos	Manage the peer assessment activity
Quickstart guide for teachers	Definitions, calculations, and examples
Contact us	eBook - 7step guide