Add-on Review Process Redesign Proposal
Revision 2

This document is a proposal to redesign the add-on review process in order to make it more efficient. It's not final and it's meant for gathering feedback from the community.

OBJECTIVE

FEATURES
#
 		Feature
 		Description
 		Phase
1.1
 Reliable add-ons
 After a number of successful reviews, add-ons can be marked as reliable, making their future reviews much simpler.
 1
1.2
 Updates for reliable add-ons
 Updates for reliable add-ons will be based on a probability system that will allow most updates to pass automatically.
 1
1.3
 Editor tool changes
 The diff and code check tools will be modified to work with the new system.
 1
1.4
 Author reputation system
 The reputation of authors will play a part in marking add-ons as reliable. A private reputation score system will be in place to assist editors.
 2
2.1
 Admin tools for reliable add-ons
 Admins will have access to a queue for pending reliable add-on decisions, and will be able to change this flag for any add-on.
 1
3.1
 Language pack and search engine reviews
 Language packs and search engine add-ons will have a more personalized review process, given that few individuals are capable of properly reviewing them.
 2
3.2
 Theme reviews
 Themes will have a different review process that will facilitate the work of editors by generating a series of screenshots that give the editor a clear view of the themed UI.
 2
4.1
 Sandoxed add-on updates
 The update.rdf file will include a new flag that will require Firefox to prompt the user for update installation.
 2
5.1
 Binary and obfuscated add-on submission
 The submission form will make it easier to submit binary and obfuscated add-ons, so that they don't need to do so much work in order to get reviewed.
 2
5.2
 Language specific add-on submission
 Add-ons that require knowledge of a specific language to be reviewed will indicate it in the queue.
 2
6.1
 User abuse feedback
 All add-on pages will have a 'Report Abuse' link that will allow users to report serious problems in add-ons.
 1


DETAILS

1.1 - Reliable add-ons
All nominations and the first N updates should be reviewed by editors. (N ~ 5)
After the Nth update is done and the add-on has been public for at least a year, the reviewing editor will be given a choice: to mark an add-on as reliable or not.
This is a judgment call made by the editor, based primarily on the add-on's review history and the author's reputation (see 1.4). New editor guidelines will be defined to help with this decision. The editor also has a choice to delegate this decision to an admin. This is what would happen if the editor decides to ignore the decision.
If an editor decided an add-on is not reliable, the same choice will be presented again for the next update of this add-on.

Note: the term reliable is used because trusted already has a different meaning within the system. Perhaps there's a better term for it.

Result: Measurable drop in the update queue due to add-on review passthrough.

1.2 - Updates for reliable add-ons
All updates for reliable add-ons will have an X% probability of being put on hold to be reviewed by an editor. If lucky, the update will be pushed to the public immediately. (X ~ 15%)
There should be a maximum number M of 'free' updates a reliable add-on can get (M ~ 10). The Mth update will be reviewed by an editor.

Note: this gives the system a great deal of flexibility, and also reduces its vulnerability to gaming by introducing a randomizing factor.

1.3 - Editor tool changes
All updates will be run through the automated test tool, as done currently. If certain flags are fired, then the developer will be notified. If the developer decides to submit anyway, an editor review will be performed on this update.

The diff tool should show the diff between the new version and the version reviewed last by and editor. This ensures all code changes are reviewed by an editor eventually.

Result: Facilitates the implementation of 1.1

1.4 - Author reputation system
All AMO authors will have a private reputation score, which will assist editors solely in making a decision on whether an add-on should be reliable or not. The scoring system will range between -10 and 10, starting at zero for new authors. Possible scoring criteria include:

2.1 - Admin tools for reliable add-ons
Admins will have access to a new approval queue for pending reliability approvals. The queue will show all add-ons that are pending decision, but it will give priority to the ones recently delegated by editors. An 'admin burndown' should be scheduled in order to reduce this queue significantly.
Admins will be able to set or revoke the reliable status on any add-on.

Update: This should be a list derived from a query that pulls add-ons not designated with a reliability status and match the criteria of 5 updates & 1 year of public status - Rey 10/14/09

3.1 - Language pack and search engine reviews
Language packs and locale-specific search engines will have a subscription system and no queues. Trusted individuals (editors, known localizers) can add themselves to be notified on updates to specific language packs. Admins will be actively look for reviewers for language packs and search engines with no subscriptions.

3.2 - Theme reviews
Themes are not allowed to have code but include lots of CSS and images that make them hard to review, specially on multiple platforms. A service similar to http://browsershots.org/ and others will be implemented. Screenshots for important Firefox windows in significant platforms are automatically created and added to the review page.
After this system is implemented, update reviews should be quick and perhaps a reliable status won't apply to themes anymore.

4.1 - Sandoxed add-on updates
Users will be allowed to receive automatic updates for sandboxed add-ons they've installed. Sandboxed add-on versions will have a new tag in the update file, like <em:requirePrompt>true</em:requirePrompt>, which makes Firefox prompt the user for confirmation before downloading and installing the update. A checkbox will be included in the prompt to prevent it from appearing again for that specific add-on.

Note: This requires a change in Firefox (Thunderbird, SeaMonkey, etc.), so it may be harder to get done. It should be implemented in Firefox 3 and above, and the site change couldn't be implemented until Firefox 3 and above users are the majority. The alternative would be to allow automatic sandbox updates to all 'old' versions.

5.1 - Binary and obfuscated add-on submission
The submission form will facilitate submission for authors of binary or obfuscated add-ons. Developers will be able to indicate their condition right away and attach a zipped version of their readable sources. These submissions will be flagged for admin review right away and the extra sources will be integrated into the diff tool.

5.2 - Language specific add-on submission
If an add-on requires knowledge of a language other than English in order to be reviewed, the submission form should provide a way to indicate so. This flag will appear in the review queue so that specialized editors know which add-ons require their knowledge.

6.1 - User abuse feedback
All add-on pages will have a 'Report abuse' link featured prominently, maybe under the 'Share' link, which contacts admins about potentially serious problems with an add-on. This doesn't really affect the review process, but it will allow to users to report problems quickly given that security will be lessened with the new system.
This would be a form where the report can be detailed, and it wouldn't appear publicly on AMO.

Update: Users will be able to select from canned reasons or enter free-text reasons - Rey 10/14/09

Update: We should generate a daily report that tells us which add-ons have received the most abuse reports. - Rey 10/14/09

CHANGELOG
Revision 2: