TEAMMATES Online Feedback System for Educators 
[System Specification (Spec)]^[a]

Preliminaries

Features are organized by user. We document requirements as a hierarchical feature list.

Feature ID : We use short titles instead of numbers to identify features. For example, we can refer to a feature as ‘Instructor:ManageCourses:Create’. This also allows easy insertion of new features into the document.

{text within curly brackets are explanatory notes}

Feature priorities

• Priority high: Must have. The system will be unusable without it.
• Priority medium: Can do without, but will be very inconvenient.
• Priority low: There is an alternative, slightly less convenient way of doing it.

Feature status: [pending] Annotation is used to mark features not implemented yet but likely to be implemented in the future.

Data definitions: A data field is described under the feature in which it is populated. e.g., What kind of data is allowed as the Course Name should be defined under createCourse feature.

Last updated at: V4.74 (half-updated at V5.72)

Table of Contents

Preliminaries

Table of Contents

--------------------------------------------------------------------------------------------------

TheProduct

Vision

Identity

Users

--------------------------------------------------------------------------------------------------

CommonFeatures

Login

Logout

Website

HomePage

AccountRequestPage

FeaturesPage

AboutPage

TermsOfUsePage

ContactUsPage

--------------------------------------------------------------------------------------------------

InstructorFeatures

Homepage

ShowOnlyRecent

GettingStartedLink[pending]

HelpPage

ManageAccount [pending]

EditAccountDetails

ManageCourses

CRUD

Archive

ManageInstructors

CRUD

AccessControl

ManageStudents

MassEnroll

ViewList

SendInvite [to be removed]

AddStudent [pending]

EditStudent

MassEdit

DeleteStudent

MassDelete[pending]

ChangeTeamStructure

SearchStudents

StudentRecords

CommentOnStudents

EmailStudents

TagStudents [pending]

ManageFeedbackSessions

Create

ViewList

Edit

Delete

Clone

ManageFeedbackQuestions

Create

ViewList

QuestionTypes [partial]

Edit

Delete

Clone [pending]

Upload files[pending]

ManageFeedbackSessionReports

ByRecipient

ByGiver

AsTable

isSeenByStudent[pending]

PrinterFriendlyView

Download

EditSubmissions

Publish

Unpublish

--------------------------------------------------------------------------------------------------

StudentFeatures

HomePage

ShowRecentOnly [pending]

HelpPage

ManageAccount

ResetGoogleId[pending]

ManageCourses

Join

ViewTeams

AutoJoin [pending]

ManageFeedbackSessions

Submit

View

Edit

ViewReport

--------------------------------------------------------------------------------------------------

AdminFeatures

ManageSystem

OnlineOffline

Statistics [pending]

ManageData

PurgeOldData

CreateInstructor

EditInstructor [pending]

DeleteInstructor

FindEntity[pending]

ManageEntity

Monitor

Masquerade

ActivityLog

--------------------------------------------------------------------------------------------------

AutomationFeatures

EmailAlerts

FeedbackSesssionOpeningAlert

FeedbackSesssionClosingAlert

FeedbackSesssionResultsPublishedAlert

Other

AutoCloseSessions

AutoPurgeOldData [pending]

ContextualHelp

--------------------------------------------------------------------------------------------------

SupplementaryRequirements

PointCalculationScheme

Report Format

Scalability

Security

--------------------------------------------------------------------------------------------------

Data Validation Rules

Glossary

--------------------------------------------------------------------------------------------------

TheProduct

Vision

TEAMMATES is a tool to help instructors manage various forms of feedback in a class, including peer evaluations.

Our target users are instructors. Their students are secondary users. i.e., Instructors decide to use the system, not students.

Identity

Name : The name TEAMMATES was chosen because the original focus of the system was managing peer feedback/evaluations in team projects. Since then, our scope has expanded to other forms of feedback. However, we are going to stick with this name for now until we reach a point of 'rebranding' in the future.

Tag line : Manage feedback in your class.

Logo:  

Users

• Instructor
• Student
• Administrator {Administrator of the system}

--------------------------------------------------------------------------------------------------

CommonFeatures

These are the features common to all users.

Login

Users use Google ID to log in

Logout

Website

{'Website' refers to static pages describing our product, which is a WebApp }

• HomePage 

• An overview of the system {Has to be perfectly done as visitors get their first impression of the system from this section}
• Submission count {an estimate is shown}
• Login buttons i.e., Login as Instructor, Student
• Key strengths of TEAMMATES: Other useful information about the app to make a good impression about the system.
• Demo video

• AccountRequestPage

• A link to 'Request for a instructor account' [pending: Move from GoogleDocs form to a proper request processing system.]

• FeaturesPage

• An overview of main features

• AboutPage

• A list of past and present project members and contributors.

• TermsOfUsePage

• Privacy, Quality of Service, etc.

• ContactUsPage

• Links to bug tracker, contact email, etc.

--------------------------------------------------------------------------------------------------

InstructorFeatures

Homepage

The first page user sees upon login. Contains shortcuts to actions the user is likely to do next.

• ShowOnlyRecent

• [priority=medium]
• Show response rate for only courses within last year. This should cut-down loading time. Need to click individual links to see response rate for older courses.

• GettingStartedLink[pending]

• Show a getting started link to new users

HelpPage

• Contains things such as Getting Started, Helpful Tips, FAQ

ManageAccount [pending]

• EditAccountDetails

ManageCourses

{See glossary for the specific definition of ‘course’}

• CRUD

• Course ID is not editable.
• Allow editing course name [pending]
• When deleting, system shows what will be deleted (students, evaluations etc.), warn about irrecoverable loss of data, and requests for confirmation [pending: Currently it’s a general warning, we may want to give more details on what data will be deleted.]

• Archive

• Archived courses are hidden from some views and shown with less details in other views

ManageInstructors

Making changes to instructors in a course is done with the Add/Edit Course forms.

• CRUD

• Done through the ‘Edit Course’ page.
• Currently, instructors can remove themselves from a course. Deleting all instructors from a course is allowed too.
• When adding a new instructor, an account is created automatically.
• [pending: email new instructor an invite instead of requiring Google ID as an input]

• AccessControl

• Fine-grained access control for instructors.

ManageStudents

• MassEnroll

• Leading/trailing whitespace and blank lines are ignored. Only email needs to be unique.
• Comment is optional. It gives us flexibility to add more details e.g., a phone number. Note there is no column for Google id. The system will allow students to log in using their google id and attach their google id to their email address.
• [pending] ‘team’ column can be left blank if the student hasn’t been put in a team yet. 
• After any mass enroll action, changes to the database will be shown to the user as feedback.
• If the same student was added twice, it will be ignored with a warning (warning is [pending] )

• ViewList

• SendInvite [to be removed]

{The concept is ‘joining’ is to be removed from user-visible features.}

• Can send invitation email to all unregistered students in a class. Normally, there is no need to do this because the invite is sent automatically at the opening of the first evaluation of that course.
• Can send invitation to a single student if the student has not joined the course yet. Possible usage:

• In case a student claims he/she did not receive the invite or gave the wrong email address.
• A student was added to the course after the invite were sent out.

• AddendumToInvite [pending] Instructor can add additional information to the invitation email sent to students.

• AddStudent [pending]

• This will be done using [MassEnroll] above

• EditStudent

• A normal way to edit student details, one student at a time.

• change team anytime
• change email any time {this is the only way to change the student’s email}
• change google id anytime. [pending]

• MassEdit

• [priority=low, smaller changes can be done using ‘Edit student’ feature above. For mass changes,

• can delete and add the whole course again.
• Can be done similar to ‘MassEnroll’ above. However, any changes to email address will be treated as adding a new student.

• DeleteStudent

• A convenient way to delete one student at a time.

• MassDelete[pending] 

• Delete all students in a course. Useful for correcting multiple mistakes during enrollment.

• ChangeTeamStructure

• Changes can be done via ‘enroll’ or ‘edit student’ features. Changes are applied to all existing sessions too, even the CLOSED ones. This can create ‘orphan’ submissions. e.g., if a student was moved to another team, his previous submissions to members of the original team will be orphaned and no longer visible.
• Intended usage is

• The instructor discovers a mistake in team structure soon after opening the first evaluation
• A student joins/leaves the course midway into the course.

• PreserveOrphanedSubmissions [pending] [priority: low]
  ChangeTeamStructure feature given above can create ‘orphan’ submissions. The system preserves such submissions. If the instructor switch to the previous team structure, these submissions will become visible again. This prevents loss of data due to inadvertent changes to team structure.

• SearchStudents

• search for a student by partial name.
• list students in a course
• list students in multiple courses

• StudentRecords

• view past results for/from student
• see photo of student [pending]
• see public info about student (e.g. LinkedIn profile) [pending]

• CommentOnStudents

• add a comment about a student
• add comments to multiple students [pending]
• add comments at different visibility levels
• view comments for student

• EmailStudents

• email student while the student is in school
• email student after the student has graduated [pending]
• email a group of students filtered by a search

• TagStudents [pending]

• tag student
• tag a group of students
• tag students
• tag students at different visibility levels (?)
• search students by tags

ManageFeedbackSessions

We use ‘FeedbackSessions’ to refer to sessions with customizable questions.

• Create

• ViewList

• Similar to that of Evaluations

• Edit

• Delete

• Clone

ManageFeedbackQuestions

• Create

• ViewList

• QuestionTypes [partial]

• Essay
• MCQ
• ?

• Edit

• Custom assignment [pending] : The ability to assign which particular student should give feedback to which other student. e.g. each student is assigned 3 other students to review. {Currently, this needs to be done outside the system}
• Delete responses if editing,

• changes visibility level - e.g. a response given under one ‘anonymous’ visibility level should not be used under another visibility level.
• affects the validity of existing responses - e.g. change options of an MCQ question

• Delete

• Clone [pending]

• For reusing past questions

• Upload files[pending]

• This is useful for students to review each others work
• No plans to implement this anytime soon because of high storage cost.
• Alternative: manage file uploads outside the system.

ManageFeedbackSessionReports

All reports are viewable at any time.

• ByRecipient

• ByGiver

• AsTable

• isSeenByStudent[pending]

• Shows if the student has read the result

• PrinterFriendlyView

• Download

• EditSubmissions

• Publish

• Unpublish

--------------------------------------------------------------------------------------------------

StudentFeatures

HomePage

The home page contains things requiring student's attention e.g., evaluations to submit, reports to view etc.

• ShowRecentOnly [pending]

HelpPage

Contains details such as Getting Started, FAQ etc.

ManageAccount

• ResetGoogleId[pending]

• This is for cases where student changes primary email to a Gmail address from a non-Gmail address.

ManageCourses

• Join 

{To be replaced by AutoJoin given below}

• Use the key to register for a course

• ViewTeams

• This is to let students view team details before any evaluation starts. It will help to detect any human errors in the student enrollment process.

• AutoJoin [pending]

• The ability to automatically join when another URL is accessed together with the registration key

ManageFeedbackSessions

• Submit

• View

• To view a submitted responses [pending: currently, only available for OPEN (via edit), but should be available for CLOSED as well]^[b]

• Edit

• Only available during status OPEN

• ViewReport

• Only available during status PUBLISHED
• The report shows following fields.

• Peer feedback appear in alphabetical order of comments (otherwise, student can guess who gave which feedback).
• Also shows what team members said they contributed to the project.
• Also shows own submission: {This is for student's own reference. It is unlikely students remember what he entered but he might want to recall them after seeing ratings/comments his teammates gave him}.

--------------------------------------------------------------------------------------------------

AdminFeatures

ManageSystem

• OnlineOffline

• Ability to take the system offline/online for maintenance etc.
• Ability to set the message shown to users during downtime [pending: Currently, shows a fixed 'under maintenance message'].
• Ability to do the above without deploying a new version.

• Statistics [pending]

• View how many students, evaluations, etc.
• Currently done as a client script

ManageData

• PurgeOldData

• To get rid of old data when we reach the storage limit.

• CreateInstructor

• instructor and add some sample data for the new instructor to see.
• Send automatic email about approval. [pending]

• EditInstructor [pending]

• DeleteInstructor

• Delete instructor status only.
• Delete entire account.

• FindEntity[pending]

• To find an entity easily (like a global search)
• Other questions that are useful in answering:

• What sessions are currently open?
• Who are the recent signups?

• ManageEntity

• This will edit/delete any entity (student, evaluation, course, etc.) while taking care of cascading deletes as well.
• {Currently done using the masquerade mode}

Monitor

• Masquerade

• Ability to impersonate any user (this will be useful for testing and troubleshooting) by adding &user=userIdto the URL.

• ActivityLog

• The activity log allows the administrator to track all user activities in the system. It captures all HTTP requests to the servlets, as well as their responses. It also captures all errors and exceptions within the system. The administrator is able to filter and search through the list of Activity Logs to locate those that he needs.
• Some questions the admin should be able to answer using the log:

• What are the activities by instructors who joined recently?
• What errors were made by users in recent times?

--------------------------------------------------------------------------------------------------

AutomationFeatures

EmailAlerts

• FeedbackSesssionOpeningAlert

• forStudent:Automatically send an email alert to students when an evaluation opens. If the student is yet to join the course, send the invite with the alert. One copy of the email is sent to each instructor.
• forInstructor [pending: [priority=low]Send a similar email to instructor 24 hours before the evaluation opens. {This helps to prevent 'accidental' opening of evaluations because the instructor set the wrong opening time}]

• FeedbackSesssionClosingAlert

• forStudent: Automatically send an email alert to students who haven't submitted an evaluation at 24 hours prior to closing time. One copy of the email is sent to each instructor.
• [pending: This email should not be sent if the opening alert was sent within last 24 hours (i.e., within 48 hours of the closing time)].
• forInstructor [pending: Automatically send an email alert when the evaluation closes.]

• FeedbackSesssionResultsPublishedAlert

• Automatically email students to inform that results have been published.

Other

• AutoCloseSessions

• Automatically closes evaluations and feedback sessions at closeTime (=endTime+gracePeriod).

• AutoPurgeOldData [pending]

• Periodically purge old data.
• Unlikely to implement unless we run out of space

• ContextualHelp

• Use hover-over tips to explain things.

--------------------------------------------------------------------------------------------------

SupplementaryRequirements

PointCalculationScheme

Students enter contribution estimates for self and team members, using the contribution scale (see glossary).

Based on those values, we try to deduce the student's answer to the following two questions:

(a) In your opinion, what portion of the project did you do?

(b) In your opinion, if your teammates are doing the project by themselves without you, how do they compare against each other in terms of contribution?

In the calculation, we try to avoid (a) affecting (b). We use (b) to calculate the average perceived contribution for each student.

Calculation steps:

1. calculate normalizedClaimed values. This is required because the total of points entered might not sum up to 100 x (team size).

(normalized value) = (original value) * (normalization factor)

e.g.,

entered values 90 [self], 110,130, N/A (total = 330)

normalization factor: (100*3)/(90+110+130) = 300/330

normalized: 82, 100, 118, N/A

normalized total = 300 (i.e. 100*number of inputs)

This answers the question (a) above. The student thinks he did 'Equal share - 18%' (as indicated by 82).

2. Calculate peerContributionRatio values by removing self-rating bias

Here, we ignore the self rating and normalize remaining values.

e.g.,

normalized input (from above): 82,100, 118, N/A

Calculating unbiased values:

82 → ignored.

100 → 100*200/(100+118) = 92

118 → 118*200/(100+118) = 108

Unbiased values: [self (ignored)], 92, 108, N/A

Unbiased values total = 200 (100*number of ratings) This answers the question (b) above. In the example above, the student thinks his teammates contribution ratio is 92:108 and is unsure of the third teammate.

3. Next, we calculate averagePerceivedContributionRatio among team members, independent of (a). This consists of these steps:

i. calculate averagePerceived:

For each student, take the average of peerContributionRatio that others have given him.

ii. calculate normalizedAveragePerceived:

Normalize the averages, similar to how input was normalized.

normalizedAveragePerceived = averagePerceived*normalizationFactor

normalizationFactor = 100 * (number of students with averagePerceived values)/(sum of averagePerceived)

This is the relative work distribution among team members based on unbiased opinions of team members.

iii. calculate normalizedPeerContributionRatio.

Since we normalized the averages (in previous step), we also normalize the value that were averaged in the first place. This is such that average and averaged tallies with each other.

normalizedPeerContributionRatio = peerContributionRatio*normalizationFactor

4. For each student, denormalize normalizedAveragePerceived. We scale back to match the total of original input by student. That way, student can compare his input (i.e., his opinion of the team’s work distribution) with the team’s opinion. In the example used above, we should use 330/300 as the denormalizing factor for that student. The result could be something like this:

student’s opinion: 90 [self], 110,130, N/A (total = 330)

team’s opinion : 95, 105,125, 115 (total = 440)

Value transformation steps: input (i.e. claimed) → normalizedClaimed → peerContributionRatio → averagePerceived → normalizedAveragePerceived → denormalizedAveragePerceived → normalizedPeerContributionRatio

Student view:

• for claimed contribution, show: same as what the student entered initially (otherwise, the student will be confused as to how the value got changed)
• for perceived contribution, show: denormalizedAveragePerceived

Instructor view:

• for claimed contribution, show: normalizedClaimed
• for perceived contribution, show: normalizedAveragePerceived

Note:

• Scenario 1: If students give 0 points to each other, then everyone should receive Equal Share and difference should be 0.
• Scenario 2: If students are not sure or do not submit the evaluation, then Perceived/Claimed for Instructor should be shown as N/A instead of Equal Share. In this case, difference too should be shown as N/A.

Report Format

Some important questions a instructor may want answered (i.e., the system should provide ways of answering these quickly and easily)

• This guy's average is rather high. Did everyone give him positive points or was the average skewed by his friend? {This is addressed by SummaryReport}
• This guy got very bad comments. What did he say about others (may be there were mutual bad feelings between him and the team)? {This is addressed by DetailedReport ByStudent}
• Who overestimated their contribution? {Can be done by sorting based on [claimed-perceived]}
• Who are high/low contributors? {Can be done by sorting based on perceived}

Reports have to be highly optimized for space, navigation, and attention.

• space: should not have very long reports with lot of white space. This is particularly a problem if we use table format and one cell happens to contain a large blob of text like a super long rant by a student.
• navigation: instructors should be able to find information easily without having to scroll up and down. Think of locality of information. Data that are related should appear together.
• attention: the report should highlight important things and provide ways to zoom in important things. for example, instructor should be able to find the cases that require his attention most. using colors, sorting facilities, etc can help here.

Scalability

• In phase 1, the system should be able to handle

• Classes of up to 300 students
• Up to 10K students
• Up to 100 instructors

• In phase 2,

• Up to 100K students
• Up to 1000 instructors

Security

• Keys are not too similar to each other {to avoid ‘guessing’ of keys}.
• Prevents masquerading via URL manipulation.
• Immune to DoS attacks.
• Immune to cross-site and SQL injection attacks.
• Only the person who deployed the system has the key to access the API directly.

--------------------------------------------------------------------------------------------------

Data Validation Rules

Data validation rules (e.g. max length, allowed characters, etc.) are defined in teammates.common.util.FieldValidator.java

Glossary

Refer to the Glossary in the Developer Manual

--- End of Document ---