Background

Pre-Remora
Prior to the launch of AMO version 3 (Remora), all add-on submissions (new submissions and updates) were treated equally and placed in a queue for an official reviewer to address. Many add-on developers were not happy with this system because their new add-ons and updates were completely inaccessible to the public until a reviewer approved them.

Remora and the Sandbox
The "Sandbox Model" was announced almost 3 years ago to address concerns with the previous model and was implemented with the launch of Remora in March 2007. Under this new process, new add-on submissions and updates to existing add-ons were treated separately, and these unreviewed add-ons were placed in a "sandbox" where the public could access them if they logged into an account and had enabled the sandbox.

The role of "reviewer" was changed to "editor", the idea being that everyone could be a reviewer by trying out the add-ons in the sandbox and reviewing them. Once an add-on had reviews, the author could nominate it for "public" status, and the editor would look at the reviews and determine if it met the quality bar for being publicly listed.

Very few add-ons were given the "trusted" status, which would allow them to submit updates that could be marked as public without editor review.


Post-Remora
Over the years since Remora's launch, slight changes were made based on user feedback that dramatically increased the exposure of sandboxed add-ons. The first of these changes was enabling the sandbox for all accounts, followed by allowing sandbox add-ons to show up in search and browse results. Finally, several months ago, the requirement to be logged-in to download a sandbox add-on was removed and replaced with a simple checkbox to ensure the user understands the risks of installing unreviewed extensions.

Today
Today, the two primary complaints AMO receives related to this problem are the time it takes to review an add-on update, and that updates to add-ons in the sandbox are not pushed out to users.


Objectives

In addressing this feedback and designing a new system, we want to:

• Make all add-ons more accessible and discoverable in a way that respects the security issues of unreviewed code
  
• Make add-on updates less painful for both developers and users
  
• Reduce wait time for reviews OR make reviews less important to the ultimate goal of the developer (add-on distribution)
• Get rid of the term "sandbox"
• Provide a way for add-ons that don't want to host on AMO to still be discoverable through AMO
• Support multiple release channels per add-on
  

Proposal

Part I: Trust Modifiers
Currently, the trustworthiness of an add-on is determined exclusively by editor review and the status that editor assigns to it. Each add-on on AMO already has qualities that affect our trust of that add-on and its author that can be leveraged for a more comprehensive assessment.

The add-on properties/data we have or will have that affect trust are:

• Editor Review - an editor's assessment of the add-on
  
• Active Users - the number of users who have the extension installed
  
• Ratings - the Bayesian rating of an add-on based on all user reviews
  
• Flags - the number of times a user has flagged the add-on as a violation (to be implemented)
  
• Add-on Verification Tool - automated check of add-on packaging, adherence to policies, and common security problems (to be implemented; see spec)
  
• Support Information - does the author provide a support URL or e-mail address?
  
• Other Add-ons by the Developer - how much do we trust the other add-ons this developer has made?
  

Part II: Measuring Trust
Some of the above qualities affect trust in a good way and some in a bad way, and some are more important than others. A formula can be used to calculate a "trust score" for each add-on that takes into account the above qualities, and provides room for expansion and tweaking as new qualities emerge or relative importance shifts.

Releasing this formula publicly may increase its vulnerability to gaming, but will also help us be transparent about our processes and encourage community feedback as to how we can improve the formula over time.

As this will seem very complicated to someone new to add-ons, we would start users off with basic things they can do to improve their trust and guide them as they move along. If we were to do a prototype of this system, we would likely not use all of the modifiers at first.

Part III: Assigning Functionality to Trust Score
Once an add-on's trust score has been calculated (which will be at least once per day), we can open up functionality to that add-on based on that score.


The vast majority of add-ons should be in the middle (green) range. Note that as this proposal is an overview and does not yet include formula details, the "trust score" values are arbitrary and there is nothing for comparison yet.

Using this trust score method will remove add-ons from the buckets of "public" or "sandbox" that they are currently in and give them more reason to stay active and work to maintain or improve their trust score to keep functionality. Developers won't be told their add-on is trapped in a "sandbox", immediately making them feel that they have to get out of the sandbox to be useful. Developers shouldn't feel pressured to get their add-on reviewed by an editor as soon as possible if the only difference it will make is removing a checkbox.

It is likely that modifier values would be set so that in order for updates to be enabled, an add-on must at least pass the automatic verification tool, and in order to remove the checkbox, an editor would have to review the add-on.

Part IV: New Distribution Features
As part of this new system, AMO would introduce two new distribution-related features: release channels and third-party listings.

Release channels would allow an add-on to have multiple chains of versions set up instead of one. For example:

• "beta" channel: 1.0b1 -> 1.0b2 -> 1.0 -> 1.5b1 -> 1.5b2 - 1.5 -> 2.0b1
  
• "release" channel: 1.0 -> 1.5

This is useful for larger add-ons that have their own development communities, like Firebug and Weave, but also for add-ons that want to support an upcoming version of Firefox in a way that does not support the current version of Firefox.

Third-party listings, or stub listings, are add-on entries on AMO that are not actually hosted on AMO. These entries will ping the extension's update.rdf file once a day for updated metadata, and point users to the extension's website instead of an "Add to Firefox" button. This will help make AMO the one-stop-shop for Firefox Add-ons and improve discoverability, however these add-ons would not be treated or ranked the same as hosted add-ons, as this would bypass our review/trust process.

Community Feedback

This is only a proposal for a possible direction of AMO processes. We are very interested in feedback on this. Please note that this is in its early stages, so specific details like the weighting of trust modifiers, formulas, and details on release channels and third-party hosting will come later in spec form based on the feedback given here.

We are especially interested in feedback on the following:

• What other add-on trust modifiers are there?
• Will this new system help end users? developers?
  
• Is the "Trust Score" the right way to do this?