Moderation/Evaluation Engine
Rough Outline, David Colarusso, August 2009

The Moderation/Evaluation Engine is a simple web-based tool for facilitating distributed-human-evaluation of mixed media content over the web. The alpha version should provide a testbed for comparing the effectiveness of different rubric criteria, evaluator pools, and decision thresholds. The public beta will implement and host an open source web app and functional API, aimed at providing a generalizable tool appropriate for use in content moderation and evaluation.

What Does the Engine "Do?"

The engine allows users to define groups around a set of qualitative evaluation criteria. Once established, content can be added to each group, and the job of evaluation can be farmed out to a subset of members. In accordance with a predefined quantitative "tipping point," the cumulative evaluation of members works to categorize content (e.g., as appropriate or inappropriate, legal or questionable, spam or not spam, etc.). Additionally, it does so in a manner which is transparent--fostering public trust. The idea here is to do one simple thing well.

Features

The Engine can best be described through its functionality. Below you will find the Engine's primary features as they might appear in API documentation, minus specific variable names and syntax. Note: most of the parameters described below would be simple text strings. However, the "rubric" and "tipping point" parameters under the "newgroup" method will require their own syntax to allow for conditional statements and evaluations tied to user-input values.

The final product should consist of two interfaces accessing a common database: one a web-based GUI, and the other an http-based API. API calls should be along the lines of:

http://www.example.com/method/?parameters=values

They should produce appropriate machine parsable output, preferably in several formats (e.g., XML, JSON, etc.)

Unless otherwise noted: (1) a method's success is contingent upon receiving the full set of correctly formed input parameters; (2) valid user accounts must possess at least one verified email address; (3) valid groups must possess a verified public contact, and (4) the addition of a "format" parameter to any of the below should be allowed to provide for multiple output formats.


method
 		parameters
 		output/outcomes
newuser
 		username: a unique identifier.
email: the new user's email address.
password: the new user's password.
confirmation: a duplicate confirmation of the above.
 success,
poorly formed input,
password mismatch,
username taken,
duplicate email,
OR
fail



success is contingent upon: (1) properly formed input (e.g., valid email); (2) a matching pwd-confirmation pair; (3) a unique username, and (4) the absence of the input email from any other user account.
on success: creates a new user in the DB's list of users with username, email, password, and random token. Sends a welcome email to the user along with the token for email verification.
edituser
new password: a proposed new password.
confirmation: a duplicate confirmation of the above.
email(s) to add: a set of email addresses to associate with the user's account.
email(s) to remove: a set of email addresses to disassociate from the user's account. 
demographic data: demographic data which may be useful for research purposes, such as: gender, DOB, level of education, etc.
authentication: confirmation that the sender has an valid user account.
success,
poorly formed input,
password mismatch,
duplicate email,
OR
fail

requires: authentication, and any or none of the remaining input parameters listed above.success is contingent upon: (1) properly formed input (e.g., valid email); (2) a matching or empty pwd-confirmation pair; (3) the absence of the input "email(s) to add" from any other user account.
on success: edit associated user account to reflect additions and alterations.
verifyemailemail: the email to which the token was sent.
token: the token provided, presumably created by the "newuser" or "newgroup" method.
success
OR
fail


success is contingent upon: a user record possessing both the input email and token.
on success: token is changed to "verified".
mergeusersauthentications: confirmation of multiple valid/verified user accounts.
parent account: the account into which all other accounts should be merged. 		success
OR
fail


success is contingent upon: confirmation of parent and merged accounts.
on success: all instances of the merged usernames are replaced by that of the parent account. 
forgottenpassusername: an existing user's username.
email: an email associated with a valid user account.
success,
no such user or email,
OR
fail

requires: ONE of the above input parameters.
success is contingent upon: (1) the existence of the provided username OR; (2) the existence of the provided email.
on success: the email on record for the associated account is emailed with a reset password.
newgroup
 		group: a descriptive group name
description: a description of the group's goals and affiliations.
rubric: a set of criteria by which group members are to evaluate content. 
tipping point: the criteria under which cumulative user evaluations trigger a status change for an individual piece of content. (This can only be entered after a rubric is entered, as the rubric may provide for variables upon which the tipping point is based.)
contact: an email address through which the public may contact the group's manager.
administrators: email addresses/usernames who's owners should have administrator privileges. (By default the author is included).
evaluators: email addresses/usernames, or group names, who's owners/members can evaluate group content. (May be null).
public voting: a variable specifying if the public can see the identities (usernames) of evaluators who have accepted invites to join the group.
required demographic data: a list of those demographic data points that an evaluator must share with the group administrators in order to evaluate content.
authentication: confirmation that the sender has an valid user account.
 success,
poorly formed input,
OR
fail


success is contingent upon: properly formed input (e.g., valid email).
on success: creates a new group in the DB's list of groups with the provided parameters and a random token for validating the contact. Sends a verification email to the contact along with the token. Adds the group to the "invites" list for those users mentioned in either "administrators" or "evaluators." Q: what to do if an email is listed which doesn't yet have a user account? Add account and email?
editgroupdescription: a description of the group's goals and affiliations.
contact: an email address through which the public may contact the group's manager.
administrators: email addresses/usernames who's owners should have administrator privileges. (By default the author is included).
evaluators: email addresses/usernames, or group names, who's owners/members can evaluate group content. (May be null).
authentication: confirmation that the sender has an valid user account.		 success
OR
fail

Note: several fields cannot be edited after group creation. To do so would allow administrators to change the terms to which evaluators agreed when accepting an invite.
success is contingent upon: properly formed input (e.g., valid email).
on success: edit associated group info to reflect alterations. If necessary: (1) Sends a verification email to the contact along with the token and; (2) Adds the group to the "invites" list for those users mentioned in either "administrators" or "evaluators." Q: what to do if an email is listed which doesn't yet have a user account? Add account and email?
groupinfo
 		group: a group's name
 (group name, description, rubric, tipping point, contact),
(group name, description, rubric, tipping point, contact, evaluators) 
OR
fail


success is contingent upon: a matching group.
on success: return information about the group consistent with its settings.
userinfo
username: an existing user's username.
(username, list groups with public voting, Gravitas rating)
OR
fail


success is contingent upon: a matching user.
on success: return relevant information about the user.
add
 		URL: the location of content to be added to the group below.
pingback (optional): a URL the Engine will ping after determination of the content's status.
group: the group to which the sender would like to add the above URL.
authentication: confirmation of the sender's user account.
 success,
poorly formed input,
invalid group,
OR
fail


success is contingent upon: (1) properly formed input (e.g., valid email); and (2) a valid group.
on success: the URL provided is associated with the group provided along with the pingback if provided.
groupqueue
username: an existing user's username.
list (optional): the list of groups you would like to query (e.g., invites, active, trash).
 success,
poorly formed input,
invalid username,
OR
fail


success is contingent upon: (1) properly formed input; and (2) a valid username.
on success: display the groups found in the associated user's invites, active, and trash lists, unless otherwise specified by the "list" parameter.
rsvp
accept: list of groups to move to the user's active list.
decline/remove: list of groups to move to the user's trash list.
authentication: confirmation of the sender's user account.		 success
OR
fail


success is contingent upon: (1) properly formed input; and (2) a the presence of the groups withing an existing list.
on success: amend user's lists accordingly. 
contentqueue
 		group: the group who's content the sender would like to explore.
status (optional): when defined only results with this status will be returned.
user (optional): the username of a user who has accepted an invite to the above group. When defined, the results will present this user's content queue of pending, evaluated, and skipped content; otherwise, it will display a queue of the group's pending and evaluated content.
 content list with classification
(e.g., pending),
invalid group,
invalid user,
OR
fail


success is contingent upon: (1) properly formed input AND; (2) valid group PLUS OPTIONAL; (3) valid user.
on success: return appropriate group or user information.
vote
 		URL: the location of content to be evaluated.
group: the group within which this evaluation should apply.
vote: the sender's evaluation (e.g., pass/fail)
authentication: confirmation of the sender's user account.
 success,
invalid group,
invalid permissions,
OR
fail


success is contingent upon: (1) properly formed input; (2) valid group, (3) user present in group's admin or evaluator list.
on success: run test for tipping point, update content status and pingback as neccesary. Calculate user Gravitas rating and update.
status
 		URL: content's location.
group: the group who's status you would like to know for the above content. 		the content's current status
invalid group,
content not present,
OR
fail


success is contingent upon: (1) properly formed input; (2) valid group, (3) valid content.
on success: return content status.