Making Open edX a Thriving Open Source Project 

by Nate Aune, Appsembler

commissioned by Stanford University’s Office of the Vice Provost for Online Learning

May 2014

Executive Summary

The Open edX project is very young -- less than a year old at the time of this writing.  No longer just the software behind the edX.org website, it now enables MOOC-scale courses for a range of schools and companies.  Organizations and individuals have begun making improvements and sharing their work, most notably Stanford, Google, and MIT.  But open-source adoption is at risk of faltering.  Open edX should open up its governance, make technical changes, and improve community management to expand beyond early-adopters.  If done right, Open edX could become the Wordpress of the MOOC space: dominant, ubiquitous, and the de-facto choice for putting your course online.

There are lessons to be learned from other successful open source projects.  Wordpress was built on dedicated community development and evangelism.  Another success story is Docker, an open source software project that launched around the same time as Open edX, and has grown at an impressive rate.  The success of Docker can be attributed to the concerted effort on the part of the Docker team to make the community development model central to everything they do. They boast an impressive 93% contributions from non-Docker staff. ^[1]  Finally, the Canvas LMS is a model that cannot be ignored.  Canvas open-sourced its software in 2011, and has strong adoption in higher education in the three years since.

There are a set of best practices that most open source projects adopt in order to ensure that the community grows and attracts the kinds of contributors that it needs to mature into a healthy and active open source community.  When done correctly, this drives a virtuous cycle: more adopters drives more courses, more course content brings users, and more users drives further adoption.  This set of best practices is the focus of this document and drive the recommendations for improvement to Open edX.  These recommendations fall into three main categories: Governance, Technical Improvements, and Community Building.  These are summarized below.  For each one there is a date that we’d like to see these changes made.  While we feel these are reasonable dates, they are Stanford’s and not committed dates from edX (they weren’t consulted on dates).

Governance

Technical Improvements

Community Building

The release of the edX platform as open source software is a key competitive advantage over other MOOC providers  New ways of integrating into other systems will emerge.  We will get code contributions from many open source developers -- what they develop will be in their own self interest, but by sharing, will benefit all.  The platform will be adapted to many more use cases than edX.org could ever do themselves.  For example: small instances instead of big; different languages and cultures; commercial applications instead of schools; SPOC’s (on-campus) vs. MOOC’s.

A particular beneficiary of open-source will be educational research.  Researchers need tools to conduct experiments, transparency to understand the systems, and ready access to data.  Usable A/B testing features are arriving soon, but some experiments will need core code changes.  I expect researchers would add many features for accessing and analyzing data.

As enhancements are contributed back to the codebase, it becomes a more featureful platform and will attract more content (course material) and that drives users (students).  This “flywheel effect” could drive transformative growth.  But this won’t happen on its own.  There are currently process and technical gaps hindering contributions and discouraging a lively community.  These recommendations are intended to address these gaps and help ensure the long-term viability of edX as an open source project with a thriving, supportive community.

Background

The edX platform was released as an open source software project on June 1, 2013.  Since that time, there have been some significant milestones in the development of the project:

1. A standalone Open edX team has been established

edX recently created a standalone team to focus on development of the Open edX software project and community.  This team consists of Ned Batchelder, James Tauber, and David Baumgold.  This is a strong initial team, and is an important signal to the emerging open source community that edX is committed to the success of the Open edX project.

2. An initiative is underway to attract and support hosting providers

edX has created a program to attract and support hosting providers.  The goal is to publish referrals to those providers for organizations seeking production level Open edX hosting.  Availability of hosting providers is critical to the long-term success of Open edX.  There are many colleges, universities, and other organizations interested in using the Open edX platform who do not have the resources in-house to stand up and support a local instance of the platform.  In addition, many faculty would create a course using Open edX if an instance were available to them for use.

3. There has been significant progress on creating API’s and Extension Points

A primary focus of the Open edX project over the past year has been to provide ways of extending and interoperating with Open edX.  This effort is critical as it reduces the need to change the core code of the Open edX platform itself.  Changing the core will always be difficult, and as the amount of code increases, so will the level of difficulty.  With these interfaces, it will be much easier for users of the platform to accomplish their pedagogical goals.

• LMS support for XBlock was recently released.  Historically, XBlocks have only been available as a developer prototyping tool, and couldn’t be used in a live course.  There is now support for them in the product.  In addition, a live course is currently running with an XBlock - one that was developed outside the core edX development team.

• Technical documentation and examples are being written to make it possible for potential platform expanders to use the interfaces that exist today.  Those are: JavaScript Display and Grade (a.k.a. “jsinput”), Learning Tools Interoperability (LTI), XBlocks, and external graders.  

• Interfaces are being created by the edX Solution and Mobile Teams to open up parts of the system.  While these are originally being created for specific customer needs, the team is making sure that all those API’s are generally available as well.  This will enable people to build alternative interfaces, like mobile clients or alternative course user experiences.  

• An initial version of an A/B Testing component was recently released.  This will enable experiments to be conducted directly through the platform, rather than requiring custom coding.

These milestones are significant accomplishments by the edX team, and they should be congratulated for them.  But there is still work to be done to foster a more widespread, community-supported open source project and platform.

This study was commissioned by the Office of the Vice Provost for Online Learning at Stanford University, and conducted under the direction of Sef Kloninger.  The purpose of the study was to collect data both from a) comparable open source projects as well as b) interviews with stakeholders of the Open edX platform, and to provide recommendations based on the data collected.

The study was conducted during the two month time period of Feb. 1, 2014 to March 31, 2014. The preliminary findings were presented to the edX organization Feb. 25-27, and their feedback was incorporated into this document. The list of stakeholders who were interviewed can be found in Appendix A

Recommendations

Recommendation for improvements to the Open edX project fall into three categories: Governance, Technical Improvements, and Community Building.  The recommendations are listed next in summary form, and then expanded on individually with specific suggestions for implementation, along with a suggested timeframe.

Governance

• Clarify and communicate the mission of Open edX
• Establish clear guidelines for contributors
• Expand governance to involve community in technical and product decisions.

Technical Improvements

• Open up the development process: public wiki’s, public bug tracking
• Move to 2-4 stable releases per year: release notes, upgrade scripts, improved packaging and testing
• Provide more ways to extend and modify the platform without having to change the core: content interfaces and API’s
• Improve the Open edX documentation
• Create a more informative website targeted at platform adopters
• Establish an ecosystem of commercial vendors and hosting providers

Community Building

• Hire a full time Open edX community manager
• Establish, measure, and communicate Key Performance Indicators (KPIs)
• Create forums to engage platform users (developers, hosting providers, researchers), e.g. user group meetings and office hours

1. Governance

“Without a way for us to be active contributing authors, it means that we’d be selecting a platform where we had no say over its future direction.”

- Lois Brooks, Oregon State University

1.1 Clarify and communicate the mission of Open edX (Q2 2014)

As many of the contributors to open source projects are volunteers, they get involved for reasons besides monetary gain. They are dedicating their time to a worthy cause because it’s something they believe in, and something they want to support. They want to feel that they belong to a community that is changing the world for the better.

The sense that I got from the interviewees was that they weren’t really sure what the mission of Open edX was.

• What change does it want to affect in the world? Why does it exist?
• What is the eventual outcome if Open edX is successful?
• Who can get involved with Open edX?
• Who does Open edX benefit and why?

edX has not clearly articulated the mission of the open-source project itself, nor how the open-source project fits into and furthers the edX mission. The former is what would inspire someone to work on it. Something like:

"Become the ubiquitous platform that all organizations (big and small, commercial and non-profit) can use to create and deliver their own world-class educational materials online."

The latter is what would justify further investment in the work. The edX mission is: "Our mission is to give a world-class education to everyone, everywhere, regardless of gender, income or social status". How can you do this with just one website?

1.2 Establish clear guidelines for contributors (Q2 2014)

Contributors are individuals who contribute to the Open edX software source code but do not have write access to the source tree. Contributions can be in the form of source code patches, new code, or bug reports, but could also include documentation.

Open edX has seen some contributions from external parties, but it’s also been observed that many are taking the code, forking it into their own repo, and not contributing these changes back.

There could be technical reasons for not contributing back. The would-be contributor could be unclear about how to contribute code that has modified the core codebase, so it’s just easier to work on a fork rather than trying to push these improvements back to the edX codebase.

But would-be contributors are also asking more process-oriented questions like:

• Who is going to look at my contribution?
• How long will it take for them to review and provide feedback?
• Is there anyone else who is making a similar feature or already fixing this bug?
• How are my contributions going to be used?
• How can I be sure that edX will be a good steward of my code? There is no visible assurance that contributions will be managed in such a way that they will remain of value to the original contributor.

When talking to these contributors, I heard concerns that it was not clear how to contribute to the Open edX codebase, especially around the process for pull requests.  It’s paramount to make very clear guidelines for contributors, so they know what to expect.

Specific recommendations:

• Establish clear guidelines for how pull requests are reviewed (perhaps delegate more responsibility to competent community members)
• Establish a framework team or technical advisory committee who are competent enough to review new features proposed by community members and deem whether they will be included in the next release or not. Make this process transparent so contributors can see why a feature was or was not accepted.

1.3 Expand governance to involve community in technical and product decisions. (Q4 2014)

There are many governance models that an open source project can adopt. Right now, it’s not clear what governance model Open edX is using, and this creates unnecessary confusion.

In regards to the public roadmap, many expressed concern that the roadmap seemed to be an announcement of what edX was planning to build, not a dialog about what features are needed by other organizations using Open edX.

There needs to be a way for institutions who’ve adopted the Open edX platform to make their voices heard, and have some influence in the direction the software is going. While they may not be contributing financially, they are contributing, in the case of Stanford, with significant developer resources and new feature development.

There seems to be a lack of transparency about how decisions are made, and this creates an environment in which outside contributors feel left out of the process.

Specific recommendations:

• Choose a governance model and communicate this clearly to the community.

• ex: Taverna Governance Model. http://www.taverna.org.uk/about/legal-stuff/taverna-governance-model/
• ex: Ubuntu Governance Model. http://www.ubuntu.com/about/about-ubuntu/governance
• ex: Docker Governance Advisory Board: Draft Proposal Version
  https://www.docker.io/governance/
  why an advisory board? http://blog.docker.io/2014/04/docker-open-governance-advisory-board/

• Establish a community council comprised of contributors who can be the voice of the community and help shape the future of the Open edX project.

2. Technical Improvements

2.1 Open up the development process: public wiki’s, public bug tracking (Q2 2014)

Right now, much of the information about Open edX exists as “tribal knowledge”, and lives in private edX wikis, private Hipchat channels and private bug trackers. This information made inaccessible to open source contributors means that they’re at a disadvantage in being able to fully participate in the project. For more seasoned open source developers this may be a deterrent to even getting involved in the project in the first place.

Make the Issue Tracker, Wiki and Hipchat Channels Public

One key turnoff for would-be Open edX contributors is the lack of a public issue tracker. While it’s understandable that edX would want to keep sensitive information out of public eyes, there really is no excuse for running an open source project without a public issue tracker.

Not does it convey an attitude of secrecy, but it causes duplication of effort because outside contributors are submitting bug reports for issues that have already been reported in the internal issue tracker.

Guarding this information behind a login screen is sending a message to the open source community - “You’re on your own to figure out that bug, because our bug tracker is not accessible to you.”

Were edX to open the issue tracker, it would potentially welcome many more contributors who could track down these bugs and fix them. But without knowing what bugs are outstanding, a contributor is left to finding these bugs on their own, bugs which may not necessarily be the highest priority.

OpenHatch is an organization that provides opportunities for people to build new skills by getting involved with open source projects and contributing to them. The OpenHatch site crawls these open source projects’ bug trackers and sends students on “missions” to fix these bugs.

edX is missing out on attracting these kinds of community activities because the bug tracker is not publicly accessible.

For every month that goes by, there are more bugs accumulating which will only make it harder to do later. More importantly, every month there are developers evaluating Open edX as a potential platform. Who knows how many run away when they discover there is no public issue tracker and deem it not a mature open source project.

This presumes that edX has fixed the process for integrating pull requests - we don't want people fixing bugs only to have the pull requests molder, or worse, ignored because someone inside edX is working on same bug.

I recommend to assign someone at edX to go through the existing private JIRA issue tracker and remove all sensitive information so that the tracker can be made public and accessible to all.

Similarly, the Confluence Wiki should be made public, or else move all relevant content to the Github wiki. The Hipchat channels should be made public, or else move all conversations to the public IRC channel.

2.2 Move to 2-4 stable releases per year: release notes, upgrade scripts, improved packaging and testing (Q3 2014)

The edX organization releases code to their instance of edX every week.  This is appropriate for them, so they can innovate rapidly, but is too difficult for someone tracking the project.  Like most projects, adopters need fewer releases with well-defined (and tested) upgrade paths.  They should have some sane naming/numbering system, and ideally be on a reasonable cadence, such as quarterly.

Lack of named releases makes it difficult to describe one’s system. It is harder to get help, harder to engage with others. This contributes to the feeling of being on your own, since each installation is custom.

They find themselves asking questions such as:

• What changes to the code occurred between the code I checked out last week, and the code I’m pulling down today?
• Will any of those changes break my site?
• How will I know if a feature that I contributed to Open edX will be made available in the next release?
• How do I plan resources to do upgrades when there is no release schedule for major versions of the software?
• What if I write documentation for a feature, and then next week that feature has changed?

It is recommended to establish a system of periodic stable releases to address this problem.

2.3 Provide more ways to extend and modify the platform without having to change the core: content interfaces and API’s. (Q3 2014)

One recurring theme among stakeholders was an observation that the Open edX software seemed to be more technically complicated than it needed to be. This complexity causes problems when trying to get newcomers quickly up-to-speed. It also makes it difficult to make small changes without needing to understand the whole system.

Other popular open source projects have grown over the years by limiting what goes in the core software, and building a pluggable architecture that allows for outside contributors (who don’t have committer access) to develop new functionality without needing to touch the core codebase.

By keeping the core software small and modular, it allows innovation to occur with add-ons and empowers the user to shape the platform to one’s particular needs.

Several of the stakeholders interviewed cited that Open edX would benefit from having a smaller core, and componentize the functionality to allow for alternative solutions. Forums is a good example of functionality that is baked into core, but really should be pluggable so that organizations could swap out the built-in forums with whatever forum best suits their needs.

2.4 Improve the Open edX Documentation (Q3 2014)

Many of the interviewees expressed frustration when first trying to get started with Open edX because there seemed to be little to no documentation. And what documentation could be found was scattered across multiple locations.

The documentation for an open source project can make or break the project. If the documentation is incomplete, hard to find, difficult to understand or non-existent, this poses a serious obstacle for the adoption of the software.

Just as there are getting started guides for instructors and course authors, there needs to be getting started guides for developers and site operators.

For Developers

Developers need better documents, tutorials, and examples about extending edX.  There is a recent push to do this: edX has a tech writer documenting the options for XBlocks, Javascript Display and Grade, and external graders.

There is also a discovery problem -- how to find what others have built?  The community would benefit from some central listing of existing modules for edX, and a way for people to indicate which bits of functionality they’re working on to reduce duplication of effort.

For site operators

edX is competing with VC-backed companies which are delivering MOOC platforms in the SaaS model meaning that there is no software to install, no servers to procure. Until edX has a comparable offering, there needs to be detailed instructions for how to stand up Open edX in production.

While edX has made the ‘configuration’ repo available which contain Ansible playbooks and Cloudformation templates, there isn’t really a step-by-step guide to deploying a production edX instance on AWS.  

The provided AMIs^[2] lower the barrier to entry, but still assume that the evaluator knows what an AMI is and how to use it. Some basic instructions would go a long way in helping evaluators get up and running with Open edX more easily and with less hassle.

Unified documentation directory with search

Right now the documentation is scattered across multiple sites. Having one site that aggregates all of these docs and provide a global search would be hugely beneficial to reduce the amount of hunting and scouring that’s currently necessary to find these nuggets of information.

2.5 Create a more informative website targeted at platform adopters (Q3 2014)

The current code.edx.org website needs to clearly communicate why someone should choose Open edX over a competing platform.

• What is the mission of the Open edX project? Is this something I believe in?
• What are the advantages of being an open source platform? Spell these out in big bold statements.
• What standards does Open edX support? List them and provide links to examples to prove it: LTI, authentication, MOOCdb, etc.
• What advantages does Open edX have over other platforms when it comes to accessibility of learning analytics data?

The website should also speak to the different types of audiences that might be looking to adopt Open edX: decision-makers, instructors, technical people. Each of these groups is expecting different entry points to begin exploring Open edX.

The decision maker wants to know who else is using Open edX, how much it will cost to run it and support it, what is the feature set. Case studies, testimonials and pricing breakdown are key things the decision maker is looking for.

The instructor wants to know whether Open edX is being used to offer online courses similar to the one they are teaching, and if it has the capabilities necessary for their class. The instructor might want to look at a sample course from a course author’s perspective, to see what the authoring environment is like.

The technical person is looking at what is required to install and configure the software, what skills are necessary to support it and extend it, what the support community is like should they have questions.

My recommendation is to create an Open edX community website focused on serving the needs of people evaluating the platform. Here is a proposed content outline for the website:

• Why Open edX?
• List of Open edX features (with screenshots and screencasts)
• List of built-in XBlocks/XModules and community contributed ones (JSDraw, etc.)
• List of other organizations who are using Open edX with links to case studies

• ex: http://plone.org/support/sites

• Getting Started - painless entry points for exploring Open edX

• Take a class (edX 101)
• Try authoring a class on the sandbox site
• Get your own demo site hosted in the cloud
• Download an evaluation version with point-n-click installer or LiveCD
• ex: Instructure Guides http://guides.instructure.com/ 

• Case studies (“MongoDB and Stanford are already doing it! you can do it too!”)
• FAQ (addressing most common objections, licensing, copyright, Stanford’s involvement)
• Readiness survey (see http://foundation.force.com/survey for example)
• How do I get involved (contribute code, write documentation, translate, marketing, etc.)

• ex: Opencast get involved page: http://opencast.org/communications
• ex: Docker community page: https://www.docker.io/community/
• ex: Plone community page: http://plone.org/community

• Blog posts with use cases (get people from the community to write them!)
• Contributor of the week (feature active contributors and show that they’re appreciated)
• Support (where can I find support? both community as well as commercial)
• Hosting (where can I find hosting? cost expectations, DIY hosting instructions)
• Standards (LTI, MOOCdb, what is XBlock and who else is using it? -> Google)

Armando Fox says there’s a need for “A ‘Demo course’ actually showing interesting features in-situ.  We have one at Berkeley that has one unit showing how to embed piazza via LTI, one unit showing how to do a simple xblock, a jsinput problem, etc.  Basically a developer sandbox but with a course that uses non-trivial features, so developers can see live examples of how to do that stuff.”

Sef Kloninger says, “We've spent a fair bit of time working on one here at Stanford actually: http://demo.class.stanford.edu/. Our course operations staff have a lot invested in it because for them their reference implementation.  Especially for advanced features like conditionals content, A/B testing, instant hangouts, LTI integration (eg. Piazza), etc.”

2.6 Establish an ecosystem of commercial vendors and hosting providers (Q4 2014)

As an open source project matures, it tends to nurture an ecosystem of third party vendors who can provide support, hosting, training and consulting around the product. If such an ecosystem doesn’t form, there’s a big risk that the product will never gain widespread adoption because it’s seen as being too risky.  Questions that arise among would-be adopters are:

• Will I have to hire full-time DevOps staff to manage the servers to run this software?
• Where can I find support if there are problems with the software?
• If I don’t like the vendor that I have, are there alternatives?
• If I don’t want to host it myself, are there viable hosting providers who can do it for me?

Anyone seriously evaluating Open edX and considering adopting it is going to be looking the availability of vendors as one of the primary selection criteria. It is recommended to do the following:

• Establish a partner program and invite capable vendors to participate

• Publish vendor guide - how to engage with edX? What can I expect as an Open edX vendor?
• Enlist vendors who can provide Open edX consulting, development, support, training, hosting
• Create a directory of vendors

• ex: http://plone.org/support/providers

• Attract hosting providers to invest in Open edX as a supported platform

• Clear up confusion about the future of mooc.org so that would-be hosting providers don’t feel like they’re going to get squashed by the Google juggernaut.
• Create Debian/Ubuntu/Redhat packages using Omnibus/fpm^[3] to make it easier for hosting providers and system administrators to install/maintain Open edX
• Use Packer.io^[4] to create Open edX images for Amazon EC2 (AMI), DigitalOcean, Docker, Google Compute Engine, OpenStack, VMWare.

3. Community Building

“Building the community with the code, is really really really important. It is harder to onboard community after there’s a lot of code, in particularly if that code is running at a pace that the community just can’t get it’s head around, and feel bought into, and the authority structures have nothing to do with the community.”

        - Brad Wheeler, Indiana University

3.1 Hire a full-time Open edX community manager (Q3 2014)

The edX community manager was very active during the initial push to open-source the project, but less so since then.  Many persons I spoke too felt that the open source community was not being given sufficient attention in order to grow a sustainable open source project that is capable of attracting active outside contributors.

I recommend the hiring of a full-time community manager whose only job is to service the Open edX community. This person might work on the following things:

• Invite people from Open edX community to regular Hangouts to talk about what they’re working on. Make it welcoming to newcomers who are curious. Invite and feature guests.

• Create an Open edX Google+ community to share links and other resources. See OpenShift as a good example: https://plus.google.com/communities/114361859072744017486 

• Open edX newsletter (subscribe to Google alert for “Open edX”)

• see Docker.io newsletter

• Call for sub-communities to form around installers, translations, learning analytics, XBlocks, etc. Self-organizing but with a clear sub-community leader who’s responsible for scheduling and documenting meetings. See Jono Bacon’s Art of Community for recommendations.

• ex: http://plone.org/community/teams

• Organize Open edX meetups - for users and developers (not just students).

• ex: https://www.docker.io/meetups/
• ex: http://plone.org/support/user-groups
• encourage people to step up and organize the meetup: https://www.docker.io/meetups/organize/

• Open edX tuneups - weekly online meetings to enlist community members to squash bugs.

• ex: Plone tuneups: http://plone.org/events/plone-tuneups

• Register Open edX on OpenHatch (http://openhatch.org/) a free service that matches contributors with open source projects

• “Superstar” award to most helpful person in IRC channel

• ex: http://plone.org/foundation/2009-help-channel-superstar

• Organize book sprint to bring together experts from the community to produce a book in 3-5 days.

• I know Adam Hyde, the founder of www.booksprints.net and www.flossmanuals.net and he connected me with...
• Anne who organized the OpenStack book sprint. Here is her proposal: https://wiki.openstack.org/wiki/Documentation/OperatorsBookSprintProposal
• Hear from book sprint participants: ex: http://justwriteclick.com/2011/10/21/google-summer-of-code-doc-summit-stories/
• OpenStack book sprint created an O’Reilly book in 5 days
• http://opensource.com/business/14/4/oreilly-openstack-operations-guide

• See the finished product here: http://docs.openstack.org/ops/

• Organize an Open edX 1 year birthday event for the wider community of open source contributors

• ex: http://www.meetup.com/Docker-meetups/events/166783242/

• Organize a strategic planning summit to bring together stakeholders and align objectives

• ex: http://plone.org/events/2008-summit

3.2 Establish, measure, and communicate Key Performance Indicators (KPIs) (Q3 2014)

You can’t improve on what you don’t measure. The Open edX project has been growing but by what metrics? We can look at many indicators to see that there has been growth, but if we’re only looking at vanity metrics, then we’re missing the more subtle aspects of growing an open source community.

Total number of mailing list subscribers may demonstrate popularity, but only when we look at how many of those subscribers are active, do we better understand levels of engagement within the community.

The same goes for contributors. Open edX may have a lot of contributors, but how many of those contributors are actively committing to the codebase versus committed once or twice?

Similarly, how many of the contributors are employed by edX or it’s partner institutions versus being truly external contributors from the community?

Open edX may have been downloaded thousands of times, but how many of those who attempted to install it were successful? What’s the failure rate for Open edX installations?

We suggest identifying key performance indicators (KPIs)  to measure Open edX progress, and set realistic targets to review monthly among edX staff and share quarterly with the community.

Proposed Open edX KPIs:

• total # of live production installations, # of courses on those installations, # of students served (these #’s could be obtained by building a “phone home” feature to auto-register a new Open edX site in network)

• See how Oscar Commerce does this by serving a single pixel image from their servers. ^[5]

• # of active members on edx-code mailing list
• # of active contributors to Open edX codebase
• % of community contributions to edX staff contributions (Docker has a whopping 93% of contributions made by community members not employed by Docker)
• # of bugs fixed by community members
• # of pull requests that get merged.  this implies that in the soon-to-be hosting ecosystem, there's a process for the hosting providers to know which release they're on, which patches/pulls they've applied, etc.
• # of edX user groups around the world
• # of talks given at conferences/meetups about Open edX

Finally, a provocative idea: what if edX redefined it’s high-level corporate goals not in terms of it’s own destination site, but in terms of any edX platform site?  The counts of number of students who have gotten statements of accomplishment, hours on site, countries served -- those are all important goals that drive the wider edX community.  Why not include all instances?

3.3 Create forums to engage platform users (developers, hosting providers, researchers), e.g. user group meetings and office hours (Q3-4 2014)

Among those who I spoke with, there was a real sense of aloneness. Many felt that they were trying to do this all by themselves with no one in the same boat. The reality was that there were others trying to stand up Open edX, but they just didn’t know about each other.

Connect like-minded community members

Jono Bacon describes in his book The Art of Community that groups are units of belonging. It’s hard to feel like you belong if you are among 1,000 other random people on a mailing list. But if groups of like-minded people are invited to participate in a sub-group, then people really start to connect because they have a shared interest.

I recommend that we make an attempt to connect people in the community based on common interests.  It could be:

• Functional: Documentation, Translations, UI/UX, marketing, etc.

• Geographical: Boston, France, Australia, etc.
• Educational level: Higher Ed, K-12, Continuing Ed, Professional
• Profession: Developer, Designer, Educator, Instructional Designer

People don’t necessarily need to be developers to contribute to the Open edX project. There’s a need for people to write documentation, put together marketing material, review UI/UX concepts, QA new versions of the software, help to organize meetups. There are many people who would probably love to help if they only knew what was needed. Again, the Art of Community has some really good advice for how to mobilize a community of volunteers to do great work.

Make two mailing lists for users and developers

The mailing list and Github repo are the first entry points that most people have with Open edX. While the edx-code mailing list has served as the primary communication channel for the Open edX community, many have observed that serious conversations get drowned out by the deluge of questions from new users asking questions about how to get the Open edX software running. The list has become too noisy to those who already have the software running or those who are actively developing the software.

Due to the high volume of technical questions, the list also seems inappropriate for non-technical faculty who want to ask about the functionality. Academics might be discouraged from posting because the conversations are operational and not pedagogical.

What many other popular open source projects have done is to divide the mailing list into two mailing lists, one for users and one for developers.

The developers list is where people who are well past the initial process of setting up the software can discuss in more depth how to extend the platform.

The users list is for people who are getting started with Open edX, and may not necessarily be developers but want to know about the software’s capabilities.

By dividing these lists, we can serve both audiences better than with a single mailing list that is trying to be all things to everyone.

Focus on evangelism and outreach

Open source projects are no different than commercial products in that they need to be marketed, promoted and evangelized. A concerted effort needs to be made to reach beyond the inner circle of contributors to spread the news about the software. This means giving talks at local meetup groups and conferences. These talks don’t necessarily need to be given by hired edX staff. In fact, it’s almost better if they are given by members of the community. But this means actively recruiting those persons who are using Open edX to give such talks.

Here are some other recommended marketing and outreach activities:

• Recruit members from community to form “Open edX marketing/evangelism committee”
• Allocate budget to sponsor and speak at popular open source conferences.  

• OSCON -- http://www.oscon.com/
• FOSDEM -- https://fosdem.org
• Open Source Bridge -- http://opensourcebridge.org/,
• PyCon -- http://us.pycon.org
• DjangoCon US -- http://www.djangocon.us/ and DjangoCon EU -- http://djangocon.eu
• EuroPython -- http://europython.eu
• SXSW Edu -- http://sxswedu.com/

• Produce 1 page marketing brochure to hand out at events (Creative Commons licensed and include source docs to allow for modifications and translations)
• Allocate budget for schwag - t-shirts, stickers, etc
• Participate in Google Summer of Code  (GSoC) and find mentors who can help students during summer.  https://developers.google.com/open-source/soc/

Summary

The open source release of the software was a significant milestone in the maturation of the edX platform.  It was the driving factor for participation by many new organizations, both in and out of higher education, and has already paid off with significant returns to edX in the form of new contributions to the codebase.  But there is still work to be done to ensure the long-term viability of edX as an open source project with a thriving, supportive community.  edX needs to open up governance of the open source project, make technical changes, and improve community management.  This will drive adoption.  The resulting quantity and variety of courses and sites, driven and by the Open edX platform, will truly further edX’s goal to “expand access to education for everyone.”

Appendix A: Methodology

This study was conducted during the two month time period of Feb. 1, 2014 to March 31, 2014, under the direction of Sef Kloninger at Stanford. The preliminary findings were presented to the edX organization Feb. 25-27, and their feedback was incorporated into this document.

I interviewed a dozen stakeholders to understand their experience with the Open edX platform. I also collected data about how Open edX stacks up against comparable open source projects.

Qualitative data was obtained via phone conversations with Open edX users or would-be users. We tried to hear from a breadth of views: technical vs. non-technical, those who have experience with the system already vs. more distant. Additional data was gathered from the edx-code mailing list.

Comments from the people interviewed are included when appropriate, and attributed only when I had explicit approval to do so.

[1]        Galen Johnson has operated an instance of Open edX on behalf of UNC, but is an employee of SAS.  He is not officially affiliated with UNC.

[2]        At the time, Paul-Olivier Dehaye wished it noted that these views were his and not on behalf of U Zurich in any official capacity.

Appendix B: Detailed Findings

Many of the people I spoke with cited one of the primary benefits of Open edX is the flexibility as open source software, to be able to take it in any direction and have total control over the branding and the data.

"Open edX is extraordinarily appealing as a platform in that we can bring it in-house and use it for whatever we want iterating on an open source platform and making it better."

- Elliott Visconsi, University of Notre Dame

"Open edX aligns with our strategic and philosophical interest and offers the most leverage of all the platforms we looked at."

- Elliott Visconsi, University of Notre Dame

"We wanted to use software that had proven it could scale. We needed to host it ourselves because of Swiss law. We can't take private data out of Switzerland, which means no AWS, and no Coursera. With Coursera there are two sets of forums. Everything is doubled because of student privacy."

- Paul-Olivier Dehaye, U Zurich

"AGPL is appealing because i can say to my colleagues that Harvard, Stanford, MIT are committing a lot of money. and they benefit from anything those universities contribute. And they still have the flexibility of modifying the code to meet their needs."

- Paul-Olivier Dehaye, U Zurich

"We believe in Open edX not for it's current state, but for it's potential. it's limited now, but being open source with this model of XBlocks, it has more potential than a centralized website. Coursera doesn't allow for much originality."

- Paul-Olivier Dehaye, U Zurich

"Moodle doesn't scale the way the edX can. We couldn't figure out what Coursera/Udemy were utilizing. They have their homegrown solutions. Open edX gave us access to the code and the community."

- Galen Johnson

"We want to see more experiments in blended learning. we don't see enough innovation in blending. It would be of great value to our system for Open edX to go in this direction."

- Matthew Rascoff, UNC

Q: How important is it for you that the MOOC platform that your university adopts is open source?

"I'm a great supporter of the vendor flexibility and feature flexibility. when there is so much flux in online learning and blended learning, we don't want to make a significant, long-term, vendor lock-in with one of these vendors. Making a big bet with their commercial roadmap is risky."

- Matthew Rascoff, UNC

"We really liked the idea of being a part of an active platform with similar goals. something aspirational and of a high standard."

- Anonymous

"Whether we're talking learning software or other spaces we're in, we can't afford to do the job we need to do for our university unless we look at partnership and economy of scale above the level of the university. That's the driver on open source. "

- Lois Brooks, Oregon State University

Q: Why was it important to have the software?

"The branding and the opportunity to innovate. This is a startup within a startup. Not just innovation with regard to curriculum, but also innovation around EdTech. So we felt it was necessary to have a platform of our own. Basically we're in control and we have access to the source code, we can meld it to the way we want the experience to be. Whereas if we were to go with a hosted experience like Udacity, we'd be locked in there."

- Shannon Bradshaw, MongoDB

"I gave edX the benefit of the doubt, that they really were going to open source the code and if they did, that would start to result in the code getting better, and they would eventually catch up and overtake Coursera. There are a lot more eyes looking at it."

- Armando Fox, UC Berkeley

"We're not going to lock ourselves into some vendor where our faculty are going to spend hours creating content and putting it into a platform whose future we have no control. If edX is going to be open source, if the relationship went south, we could take the code and stand it up ourselves. We wouldn't lose our investment. That was an ingredient."

- Armando Fox, UC Berkeley

“Even if we choose a hosted solution, it is preferable for that software to be open source. So if we were not satisfied with the hosting we can move to another hosting provider. I would love to see that kind of competitive hosting ecosystem for Open edX.”

- Stakeholder 4

"As with all open source, at some level we really want the non-beholdeness, control your own destiny. But we'd also really not have to run it. We'd rather have somebody else host it."

- Armando Fox, UC Berkeley

“In an academic environment, the decision/innovation in IT is usually driven by academics. Open edX offers to an academic the opportunity of being able to say "This is software used by millions of users, developed by top schools, and that is open source. Install it!". The administrator might be reluctant but has to do it. This happened in Zurich (at multiple scales) and Davis, AFAIK. This is made even easier with the one-click Appsembler install.”

- Paul-Olivier Dehaye, U Zurich

Through these conversations, I found that Open edX has gaps that are holding back its adoption, and the lack of an open community development process and transparent governance poses serious risk to the open-source project. I identified three overarching categories in which we can explore these gaps: technical, governance and community.

Technical

While the largest audience for Open edX is teachers, instructors and professors, at this point in the history of Open edX, and until the software is made more widely available, it’s being evaluated primarily by technical operators, usually systems administrators or developers.

They are basing their judgement on its technical merits:

• how easy is it to find support and documentation?
• how easy will it be to maintain and scale?
• how easy is it to extend the software?
• does it support common standards that will make it easy to integrate?
• how often do new versions of the software get released?
• what skills do I need to possess to adequately support this software?

If technical operators are unable to find adequate answers to these questions, the recommendation usually comes back to not use Open edX, or to wait for it to mature before using it.

Lack of hosting and vendor support

As an open source project matures, it tends to nurture an ecosystem of third party vendors who can provide support, hosting, training and consulting around the product. If such an ecosystem doesn’t form, there’s a big risk that the product will never gain widespread adoption because it’s seen as being too risky.  Questions that arise among would-be adopters are:

• Will I have to hire full-time DevOps staff to manage the servers to run this software?
• Where can I find support if there are problems with the software?
• If I don’t like the vendor that I have, are there alternatives?
• If I don’t want to host it myself, are there viable hosting providers who can do it for me?

Some of these concerns emerged while talking with these colleges and universities:

“On our own servers”

“There seems to be significant assumptions about infrastructure, in particular on Amazon, and seems to be gear towards a really large deployment. There is no other option to deploy other than Amazon. We thought it would be optimal for us to start out not going directly to AWS, but to deploy on our servers.”

- Anonymous

“Development focused”

“The biggest problem I have right now, is that everything seems to be very development focused - very development centric. And not as much focused on operational aspects of things. Like managing and monitoring the instances. Having a singular production platform. A lot of the pieces are still like puzzles.”

- Galen Johnson

“Cloud service”

“Frankly what we need is Open edX running as a cloud service.”

- Brad Wheeler, Indiana University

“Support ecosystem”

“From my perspective as a buyer and a user, I think it would be wonderful for there to be the kind of support ecosystem that you see with other robust open source communities, where there's a range of partnership and support offerings available from the marketplace.”

- Matthew Rascoff, UNC

“Tightly closed secret”

“I know Google is supposedly doing something with edX this year, but It’s not entirely clear what Google is doing with edX. It’s seems to be a tightly closed secret there. And there’s that whole Mooc.org thing that is pending and up and coming.”

- Galen Johnson

“Pick up the phone”

“Early on it was very difficult to get responses back. But they’ve gotten a lot better on the IRC channel. It’s getting there but it’s not like you can pick up the phone and talk to somebody.”

–Galen Johnson

Documentation

Many of the interviewees expressed frustration when first trying to get started with Open edX because there seemed to be little to no documentation. And what documentation could be found was scattered across multiple locations.

“Too immature.”

“The reason we didn’t adopt Open edX is because it wasn’t ready to be installed and used. It was too immature. Both from the assessment that we did but also from the edX team who said, ‘Don’t use it yet.’ We were looking for more documentation.”

- Lois Brooks, Oregon State University

“Lack of documentation.”

“The main thing besides that one was the lack of documentation about how to set it up. There were several moving parts.”

- Matt Hansen, Oregon State University

“A lot of tribal knowledge.”

“There is still a lot of documentation and components that aren’t quite there from an operations perspective. It is still a very young, almost alpha level solution. I know they have real-world installations out there, but they know the pieces. There’s a lot of tribal knowledge that hasn’t made it back into the documentation.”

- Galen Johnson

“Scattered documentation.”

“The documentation is very scattered. If I hadn’t been paying attention to the IRC channel, I would have missed it.”

- Galen Johnson

“Docs friendly to non-geeks”

“The biggest challenge is the relative lack of documentation that is friendly to non-geeks. As more faculty outside the technical departments start to use edX, there is not really anywhere they can go for help.”

- Armando Fox, UC Berkeley

Technical Complexity

Another recurring theme was an observation that the Open edX software seemed to be more technically complicated than it needed to be. This complexity causes problems when trying to get newcomers quickly up-to-speed. It also makes it difficult to make small changes without needing to understand the whole system.

“Too convoluted.”

“My main problem was that I thought the system was too wound together and too convoluted for successful open source adoption because it took too long before you could understand how to do anything.”

- Jim Waldo, Harvard University

“Monolithic.”

“There was a lot of extra technical complexity that doesn’t need to be in the platform. It’s just a lot of ramp-up even to being able to use the edX platform.

Even though it’s built on Django, it doesn’t really help you much because you need to learn XModule (which is now XBlock) before you can actually do something.

The system is somewhat monolithic. You have to understand a lot of different parts before you can actually make changes.

There were features that we think would be relatively simple to implement became more complicated because there are all these different systems to think about.

It’s not immediately apparent to a programmer what the benefits of having an Xblock or Xmodule. XBlock/Xmodule is a web component architecture, but there are already a lot of web component architectures.

It’s not immediately obvious what benefit you get over using this architecture over using something that’s already existing.”

- Graham Lowe, MongoDB

“In other open source projects that I’ve been involved in. We always had as a goal, the ’15 minute experience’, that within 15 minutes of downloading the code, you could make a change that was observable in the system.”

- Jim Waldo, Harvard University

Smaller Pluggable Core

Other popular open source projects have grown over the years by limiting what goes in the core software, and building a pluggable architecture that allows for outside contributors (who don’t have committer access) to develop new functionality without needing to touch the core codebase.

By keeping the core software small and modular, it allows innovation to occur with add-ons and empowers the user to shape the platform to one’s particular needs.

Several of the peopel interviewed cited that Open edX would benefit from having a smaller core, and componentize the functionality to allow for alternative solutions. Forums is a good example of functionality that is baked into core, but really should be pluggable so that organizations could swap out the built-in forums with whatever forum best suits their needs.

“Nobody is ever happy with the forum they have. No two people want the same forum.

This strikes me as something screams out for an entry point where we can connect whatever forum we want. But it is not trivial to do that and it is not something that is supported by the platform.”

“There was no small piece. It really doesn’t have a component architecture to it. It’s really an application as opposed to a component system. As an application, there are identifiable layers in the application, but in order to change anything, you need to know how to change everything.”

- Jim Waldo, Harvard University

“One huge improvement to the system that could be made, which is no small task and would be really hard at this point, would be for it to have a smaller core you could pick and choose what you want - a set of plugins that interact with the system. edX has so many features that we don’t necessarily use, but we have to pay the price for that.”

- Graham Lowe, MongoDB

“Which ecosystem does a better job.”

“The MOOC competitors will not be competing primarily on the base platform. People are going to try to build all kinds of things on top of them.  And a lot of the value comes from which ecosystem does a better job of supporting that. Which ecosystem has built more modules that I can use.  I've begun to stop thinking about features, and started thinking about APIs.”

- Armando Fox, UC Berkeley

Lack of Periodic Stable Releases

The edX organization releases code to their instance of edX every week.  This is appropriate for them, so they can innovate rapidly, but is too difficult for someone tracking the project.  Like most projects, adopters need fewer releases with well-defined (and tested) upgrade paths.  They should have some sane naming/numbering system.  Ideally, they’d be on a reasonable cadence -- quarterly?

Lack of named releases makes it difficult to describe one’s system. It is harder to get help, harder to engage with others. This contributes to the feeling of being on your own, since each installation is custom.

They find themselves asking questions such as:

• What changes to the code occurred between the code I checked out last week, and the code I’m pulling down today?
• Will any of those changes break my site?
• How will I know if a feature that I contributed to Open edX will be made available in the next release?
• How do I plan resources to do upgrades when there is no release schedule for major versions of the software?
• What if I write documentation for a feature, and then next week that feature has changed?

The comments regarding release schedules were the following:

“One concern that we’ve had moving over to the new version of Open edX, we’re immediately going to move behind because of all the commits that are happening on a daily basis. Is there thought for going to a timed release model that characterizes what the changes are, and essentially drives the open source effort forward? Where you have a better sense for what changes might be lurking there that you are about to adopt.”

- Shannon Bradshaw, MongoDB

“The fact that edX practices continuous deployment is a challenge because we have to decide if we should go through the struggle of continually updating to new releases and all the challenges that brings, or fork off to our own instance, understanding that we wouldn't getting these new updates.”

- Anonymous

“The perfect platform would be one that is one whenever I do an update, I don’t have to go and be concerned about things not working. They are still changing things under the hood. there are things that break.”

- Galen Johnson

Governance

It is critical that an open source project clearly communicates its policies and strategies to potential users and developers of the project’s outputs. A clear governance model also allows potential contributors to understand how they should engage with the project, what is expected of them and what protections are provided to ensure that their contributions will always be available to them.

In addition, it describes the quality control processes that help to assure potential users of the viability of the project. The development and communication of a clear and concise governance model is one of the most important steps a project can take towards sustainability through open development. ^[6]

If there is disagreement within the community, forking may seem to be the most efficient solution in the early stages of a project. However, this may prevent third parties from easily adopting the code, and will, over time, create significant maintenance problems for the Open edX team and the community at large. ^[7]

Without clear guidance on these matters, most people will walk away rather than join an immature project. But if those early adopters are shown that they can help to guide the project as it matures, they may decide to stay.

A single external contributor may well have a major effect on the sustainability of a project, so project initiators can simply not afford to risk losing that contributor as a result of trying to save a small amount of effort in the early stages.

Lack Of Collaborative Decision Making

From talking with leaders at colleges and universities, it became clear to me that many are concerned with the governance of Open edX, and in particular, about if/how collaborative decision-making will be made:

“Don’t have a voice.”

“There isn’t really any governance for open source users of Open edX right now.  Open edX users don’t have a voice at the edX table. And ultimately that’s going to be a really important forum for setting the big strategic direction for this project. I would want to make sure the governance is in place for both of those viewpoints to be heard. It’s important for there at least to be some forum for that conversation to happen and I don’t think that exists right now.”

- Matthew Rascoff, UNC

“Working on our own.”

“My takeaway away was that it felt that the development was geared towards some unknown, specific edX uses and goals, rather than in other open source tools in which the goal was to make a generic tool that could be adapted.

We felt like we're working on our own. We were hoping it would be more collaborative with shared goals. But we never really got there.”

- Anonymous

“Shared development.”

“Its not clear how much they want to be a project of partners versus an open project.”

- Peter Pinch, MIT

“The innovation of open source across the board is not software at all, but it's shared development. Unless edX really understands this and changes quickly, they are heading down a path to have a user base of grateful adopters.”

- Lois Brooks, Oregon State University

“No say over future direction.”

“It also increases the risk. Without a way for us to be active contributing authors, it means that we’d be selecting a platform where we had no say over its future direction.”

- Lois Brooks, Oregon State University

“Clubbiness”

“With an open source community you have to have a lot of moxy, and insert yourself in. But with the Open edX project, there has not been an open door for all the other universities to participate. There is just the ‘clubbiness’ of it.

Which to me is neither good nor bad, they’ve kind of set their own model on it. But it really did put us into more of a straight vendor buyer relationship, because it’s not an open community bringing in new contributors.”

- Lois Brooks, Oregon State University

“Take it. Fork it”

“There are very few folks who seem to be coming back. But just taking it and running with it on their own. Right now in its infancy to where that’s a real risk. Where people will take it, fork it, and do their own thing.”

- Galen Johnson

Unclear contribution guidelines

Contributors are individuals who contribute to the Open edX software source code but do not have write access to the source tree. Contributions can be in the form of source code patches, new code, or bug reports, but could also include documentation.

Open edX has seen some contributions from external parties, but it’s also been observed that many are taking the code, forking it into their own repo, and not contributing these changes back.

There could be technical reasons for not contributing back. The would-be contributor could be unclear about how to contribute code that has modified the core codebase, so it’s just easier to work on a fork rather than trying to push these improvements back to the edX codebase.

But would-be contributors are also asking more process-oriented questions like:

• Who is going to look at my contribution?
• How long will it take for them to review and provide feedback?
• Is there anyone else who is making a similar feature or already fixing this bug?
• How are my contributions going to be used?
• How can I be sure that edX will be a good steward of my code? There is no visible assurance that contributions will be managed in such a way that they will remain of value to the original contributor.

When talking to these contributors, I heard concerns that it was not clear how to contribute to the Open edX codebase, especially around the process for pull requests.

“Sociological barriers.”

“There are also some sociological barriers. It was never clear to me what the requirements were to get something pulled in. You submit a pull request. What is going to determine when that is going to be reviewed. What needs to be in the PR? Do you need to have unit tests? Do you need to have integration tests? What sort of documentation does there need to be?”

- Jim Waldo, Harvard University

“The contribution process has been shaky. It’s gotten a bit better, but it still feels like you need an inside track in order to get your pull requests reviewed. You need to bug someone personally to get code reviewed.”

- Peter Pinch, MIT

“The communication about when PRs will be reviewed is still not entirely clear. They want to make the code review process part of their agile sprints. It has plusses and minuses. Some of our stuff gets dropped out of the sprint if it doesn’t fit.”

- Peter Pinch, MIT

Community

Just as oxygen serves to incite a chain reaction when present with heat and fuel. From Matt Massie's presentation "Open Source: Community, Code and Infrastructure". ^[8]

So does a community incite a chain reaction when present with code and infrastructure.

Just as with a fire, even if you have heat and fuel, if there is no oxygen, the fire will suffocate and die. In the same way, even if you have code and infrastructure, without a community the project will suffocate and die.

The community is an essential ingredient in the making of a successful open source project. It’s not just enough to ignite the project, a community is what keeps it going, and makes it sustainable.

Jono Bacon says in his book, “Art of Community” ^[9]:                        

• A sense of belonging is what keeps people in communities. This belonging is the goal of community building. The hallmark of a strong community is when its members feel that they belong.
• Belonging is the measure of a strong social economy. This economy’s currency is not the money that you find in your wallet or down the back of your couch, but is social capital.
• For an economy and community to be successful, the participants need to believe in it. If no one believes in the community that brings them together, it fails.
• Like any other economy, a social economy is a collection of processes that describe how something works and is shared among those who participate.
• These processes, and the generation of social capital, which in turn generates belonging, need to be effectively communicated.

As Jono’s book title would suggest, building a community is more of an art than it is a science. It’s not something that you can buy nor is it something you can ignore. Like a family it must be cared for, nurtured and appreciated.

The control exerted by the project leadership is only as strong as the support the community gives that leadership. It is only through careful management of the community that open source project leaders remain ‘in power’. Clearly setting out the rules of the community in this way is an important part of this management process. ^[10]

Not Enough Community Management

The edX community manager was very active during the initial push to open-source the project, but less so since then.  Many persons I spoke too felt that the open source community was not being given sufficient attention in order to grow a sustainable open source project that is capable of attracting active outside contributors.

“They need a full-time position that has almost nothing to do with the technology. It has all to do with the building of a community. And they don’t have anybody doing that.”

 - Armando Fox, UC Berkeley

“We learned if you don’t build a community as you go, and I mean really really build a community as you go, it’s not impossible, but it’s more difficult to really have a community that is vibrant and participating and bringing gold and contributions.”

- Brad Wheeler, Indiana University

“If the coordination and the community aren’t engineered from the beginning, the heat of local clocks of people needing to get things done will cause it to run forward well-intended, and it will be very difficult.”

- Brad Wheeler, Indiana University

“Building the community with the code, is really really really important. It is harder to onboard community after there’s a lot of code, in particularly if that code is running at a pace that the community just can’t get it’s head around, and feel bought into, and the authority structures have nothing to do with the community.”

- Brad Wheeler, Indiana University

“I’m really comfortable with the community development cycle. With that said, it didn’t appear to me that Open edX has developed a community development process. It’s still pretty closed to a small group, which will limit it’s growth cycle. It’s going to be a lot slower than what we would want as opposed to having a robust community of developers around it.”

- Lois Brooks, Oregon State University

“I see edX heading down the same path of retaining the clubbiness while they train their user base to become grateful adopters, and as you know since you do open source.

When you do an open source community, it’s not about the software, it’s about the community. Software is a by-product of the community.”

- Lois Brooks, Oregon State University

“The most important thing for the company is community development. Either I’m a grateful adopter and I’ve got some kind of exit path, or I’ve got a community that I really have a lot of confidence in, where I believe our contributions are time well spent.

If we were looking at edX seriously one of the things we’d be looking at is baking in a quick exit strategy (back to those data models), understanding that if Harvard or Stanford or MIT lost interest.”

- Lois Brooks, Oregon State University

“Just going open source doesn’t guarantee success. What you need is that robust community of focused institutions and committer engineers, and also the commercial marketplaces to provide support surrounding it. Those are some of the things that will be really important for us making a long-term investment in Open edX.”

- Matthew Rascoff, UNC

“You’re not alone”

“Localized meetups even if they’re not globalized meetups could make a huge difference. This feeling that you’re not alone makes a huge difference.”

- Armando Fox, UC Berkeley

Noisy communication channels

While the edx-code mailing list has served as the primary communication channel for the Open edX community, many have observed that serious conversations get drowned out by the deluge of questions from new users asking questions about how to get the Open edX software running. The list has become too noisy to those who already have the software running or those who are actively developing the software.

Due to the high volume of technical questions, the list also seems inappropriate for non-technical faculty who want to ask about the functionality. Academics might be discouraged from posting because the conversations are operational and not pedagogical.

“In terms of the mailing list, this might be an incorrect assessment it’s mostly been about "i’m having problems getting the platform running”, as opposed to “I want to use the software to do this. i want to do this in my course, or I want to have this kind of assessment."

- Graham Lowe, MongoDB

“Like souls”

“I think segmenting the mailing list from newbies from those who’ve been using it and actively developing it might be beneficial. Perhaps it would stimulate more conversation, if people felt like there were other like souls out there.”

- Shannon Bradshaw, MongoDB

“The biggest challenge for the user community is that there isn't a really nice "where do I start" and "who do I call". It's really as simple as that.  There isn't really a forum for that. There is a developers forum, but there isn't a users forum.”

- Armando Fox, UC Berkeley

“Setting up some chat channels where people could talk over things. what features do people want? What are they working on? How do we keep from stepping on each other’s toes?”

- Armando Fox, UC Berkeley

Unclear mission and vision for Open edX

As many of the contributors to open source projects are volunteers, they get involved for reasons besides monetary gain. They are dedicating their time to a worthy cause because it’s something they believe in, and something they want to support. They want to feel that they belong to a community that is changing the world for the better.

The sense that I got from the interviewees was that they weren’t really sure what the mission of Open edX was.

• What change does it want to affect in the world? Why does it exist?
• What is the eventual outcome if Open edX is successful?
• Who can get involved with Open edX?
• Who does Open edX benefit and why?

“It’s not clear what edX intended to get out of the open sourcing of the code.”

- Peter Pinch, MIT

“I think it would be really helpful to have somebody who stood up as the spokesperson for the community who could enunciate a vision for the technology and lead somewhere.

Not a question of what’s going to happen in the next quarter, but a question of what do we see as the future of technology and teaching in this way

- Armando Fox, UC Berkeley

“One way to get an open source community really energized is to say what the crusade is all about, and then make it into a crusade. If we’re doing this to make the world a better place, here’s how we’re going to make the world a better place? Here’s how you can help. And they don’t have anybody talking about that. It’s very abstract on how the world’s going to be made a better place.”

- Armando Fox, UC Berkeley

Appendix C: Detailed Recommendations

Based on feedback we’ve received from those interviewed, and studying what’s worked for other successful open source projects, it is recommended to initially address the needs of teachers, since they are the largest category of adopters. Below is a diagram produced by Sef Kloninger from Stanford which illustrates the various adopters.

The pyramid shows the relative number of stakeholders interested in using the Open edX platform -- not to scale, of course! Teachers, at the bottom, are the most numerous and are looking to create content. Extenders build on that content by using API's to build immersive and engaging educational experiences. Operators and Extenders look to run and change the code themselves. While the least numerous and most difficult to support, over time engineers adding core features will be the most valuable people to have in our community.

Clarify and Communicate The Mission

edX has not clearly articulated the mission of the open-source project itself, nor how the open-source project fits into and furthers the edX mission. The former is what would inspire someone to work on it. Something like:

"Become the ubiquitous platform that all organizations (big and small, commercial and non-profit) can use to create and deliver their own world-class educational materials online."

The latter is what would justify further investment in the work. The edX mission is: "Our mission is to give a world-class education to everyone, everywhere, regardless of gender, income or social status". How can you do this with just one website?

Commercial vendor ecosystem

Anyone seriously evaluating Open edX and considering adopting it is going to be looking the availability of vendors as one of the primary selection criteria. It is recommended to do the following:

• Establish a partner program and invite capable vendors to participate

• Publish vendor guide - how to engage with edX? What can I expect as an Open edX vendor?
• Enlist vendors who can provide Open edX consulting, development, support, training, hosting
• Create a directory of vendors

• ex: http://plone.org/support/providers

• Attract hosting providers to invest in Open edX as a supported platform

• Clear up confusion about the future of mooc.org so that would-be hosting providers don’t feel like they’re going to get squashed by the Google juggernaut.
• Create Debian/Ubuntu/Redhat packages using Omnibus/fpm^[11] to make it easier for hosting providers and system administrators to install/maintain Open edX
• Use Packer.io^[12] to create Open edX images for Amazon EC2 (AMI), DigitalOcean, Docker, Google Compute Engine, OpenStack, VMWare.

Improve the Documentation

“Documentation quality is becoming the differentiator between project success and failure.”

-Chris McDonough, founder of the Pyramid project ^[13]

As Chris reminds us, the documentation for an open source project can make or break the project. If the documentation is incomplete, hard to find, difficult to understand or non-existent, this poses a serious obstacle for the adoption of the software.

Just as there are getting started guides for instructors and course authors, there needs to be getting started guides for developers and site operators.

For Developers

Developers need better documents, tutorials, and examples about extending edX.  There is a recent push to do this: edX has a tech writer documenting the options for XBlocks, Javascript Display and Grade, and external graders.

There is also a discovery problem -- how to find what others have built?  The community would benefit from some central listing of existing modules for edX, and a way for people to indicate which bits of functionality they’re working on to reduce duplication of effort.

For site operators

edX is competing with VC-backed companies which are delivering MOOC platforms in the SaaS model meaning that there is no software to install, no servers to procure. Until edX has a comparable offering, there needs to be detailed instructions for how to stand up Open edX in production.

While edX has made the ‘configuration’ repo available which contain Ansible playbooks and Cloudformation templates, there isn’t really a step-by-step guide to deploying a production edX instance on AWS.  

The provided AMIs^[14] lower the barrier to entry, but still assume that the evaluator knows what an AMI is and how to use it. Some basic instructions would go a long way in helping evaluators get up and running with Open edX more easily and with less hassle.

Unified documentation directory with search

Right now the documentation is scattered across multiple sites. Having one site that aggregates all of these docs and provide a global search would be hugely beneficial to reduce the amount of hunting and scouring that’s currently necessary to find these nuggets of information.

Have a More Informative Website Targeted at Platform Adopters

The current code.edx.org website needs to clearly communicate why someone should choose Open edX over a competing platform.

• What is the mission of the Open edX project? Is this something I believe in?
• What are the advantages of being an open source platform? Spell these out in big bold statements.
• What standards does Open edX support? List them and provide links to examples to prove it: LTI, authentication, MOOCdb, etc.
• What advantages does Open edX have over other platforms when it comes to accessibility of learning analytics data?

The website should also speak to the different types of audiences that might be looking to adopt Open edX: decision-makers, instructors, technical people. Each of these groups is expecting different entry points to begin exploring Open edX.

The decision maker wants to know who else is using Open edX, how much it will cost to run it and support it, what is the feature set. Case studies, testimonials and pricing breakdown are key things the decision maker is looking for.

The instructor wants to know whether Open edX is being used to offer online courses similar to the one they are teaching, and if it has the capabilities necessary for their class. The instructor might want to look at a sample course from a course author’s perspective, to see what the authoring environment is like.

The technical person is looking at what is required to install and configure the software, what skills are necessary to support it and extend it, what the support community is like should they have questions.

My recommendation is to create an Open edX community website focused on serving the needs of people evaluating the platform. Here is a proposed content outline for the website:

• Why Open edX?
• List of Open edX features (with screenshots and screencasts)
• List of built-in XBlocks/XModules and community contributed ones (JSDraw, etc.)
• List of other organizations who are using Open edX with links to case studies

• ex: http://plone.org/support/sites

• Getting Started - painless entry points for exploring Open edX

• Take a class (edX 101)
• Try authoring a class on the sandbox site
• Get your own demo site hosted in the cloud
• Download an evaluation version with point-n-click installer or LiveCD
• ex: Instructure Guides http://guides.instructure.com/ 

• Case studies (“MongoDB and Stanford are already doing it! you can do it too!”)
• FAQ (addressing most common objections, licensing, copyright, Stanford’s involvement)
• Readiness survey (see http://foundation.force.com/survey for example)
• How do I get involved (contribute code, write documentation, translate, marketing, etc.)

• ex: Opencast get involved page: http://opencast.org/communications
• ex: Docker community page: https://www.docker.io/community/
• ex: Plone community page: http://plone.org/community

• Blog posts with use cases (get people from the community to write them!)
• Contributor of the week (feature active contributors and show that they’re appreciated)
• Support (where can I find support? both community as well as commercial)
• Hosting (where can I find hosting? cost expectations, DIY hosting instructions)
• Standards (LTI, MOOCdb, what is XBlock and who else is using it? -> Google)

Armando Fox says there’s a need for “A ‘Demo course’ actually showing interesting features in-situ.  We have one at berkeley that has one unit showing how to embed piazza via LTI, one unit showing how to do a simple xblock, a jsinput problem, etc.  Basically a developer sandbox but with a course that uses non-trivial features, so developers can see live examples of how to do that stuff.”

Sef Kloninger says, “We've spent a fair bit of time working on one here at Stanford actually: http://demo.class.stanford.edu/. Our course operations staff have a lot invested in it because for them their reference implementation.  Especially for advanced features like conditionals content, A/B testing, instant hangouts, LTI integration (eg. Piazza), etc.”

Establish, Measure, and Communicate Key Performance Indicators (KPIs)

You can’t improve on what you don’t measure. The Open edX project has been growing but by what metrics? We can look at many indicators to see that there has been growth, but if we’re only looking at vanity metrics, then we’re missing the more subtle aspects of growing an open source community.

Total number of mailing list subscribers may demonstrate popularity, but only when we look at how many of those subscribers are active, do we better understand levels of engagement within the community.

The same goes for contributors. Open edX may have a lot of contributors, but how many of those contributors are actively committing to the codebase versus committed once or twice?

Similarly, how many of the contributors are employed by edX or it’s partner institutions versus being truly external contributors from the community?

Open edX may have been downloaded thousands of times, but how many of those who attempted to install it were successful? What’s the failure rate for Open edX installations?

We suggest identifying key performance indicators (KPIs)  to measure Open edX progress, and set realistic targets to review monthly among edX staff and share quarterly with the community.

Proposed Open edX KPIs:

• total # of live production installations, # of courses on those installations, # of students served (these #’s could be obtained by building a “phone home” feature to auto-register a new Open edX site in network)

• See how Oscar Commerce does this by serving a single pixel image from their servers. ^[15]

• # of active members on edx-code mailing list
• # of active contributors to Open edX codebase
• % of community contributions to edX staff contributions (Docker has a whopping 93% of contributions made by community members not employed by Docker)
• # of bugs fixed by community members
• # of pull requests that get merged.  this implies that in the soon-to-be hosting ecosystem, there's a process for the hosting providers to know which release they're on, which patches/pulls they've applied, etc.
• # of edX user groups around the world
• # of talks given at conferences/meetups about Open edX

Build, Nurture and Promote the Open edX community

Among those who I spoke with, there was a real sense of aloneness. Many felt that they were trying to do this all by themselves with no one in the same boat. The reality was that there were others trying to stand up Open edX, but they just didn’t know about each other.

Connect like-minded community members

Jono Bacon describes in his book The Art of Community that groups are units of belonging. It’s hard to feel like you belong if you are among 1,000 other random people on a mailing list. But if groups of like-minded people are invited to participate in a sub-group, then people really start to connect because they have a shared interest.

I recommend that we make an attempt to connect people in the community based on common interests.  It could be:

• Functional: Documentation, Translations, UI/UX, marketing, etc.

• Geographical: Boston, France, Australia, etc.
• Educational level: Higher Ed, K-12, Continuing Ed, Professional
• Profession: Developer, Designer, Educator, Instructional Designer

People don’t necessarily need to be developers to contribute to the Open edX project. There’s a need for people to write documentation, put together marketing material, review UI/UX concepts, QA new versions of the software, help to organize meetups. There are many people who would probably love to help if they only knew what was needed. Again, the Art of Community has some really good advice for how to mobilize a community of volunteers to do great work.

Make two mailing lists for users and developers

The mailing list and Github repo are the first entry points that most people have with Open edX. The problem many encountered with the mailing list was that it was too noisy. What many other popular open source projects have done is to divide the mailing list into two mailing lists, one for users and one for developers.

The developers list is where people who are well past the initial process of setting up the software can discuss in more depth how to extend the platform.

The users list is for people who are getting started with Open edX, and may not necessarily be developers but want to know about the software’s capabilities.

By dividing these lists, we can serve both audiences better than with a single mailing list that is trying to be all things to everyone.

Conduct Open Development so that all Contributors Have Access to Knowledge

Right now, much of the information about Open edX exists as “tribal knowledge”, and lives in private edX wikis, private Hipchat channels and private bug trackers. This information made inaccessible to open source contributors means that they’re at a disadvantage in being able to fully participate in the project. For more seasoned open source developers this may be a deterrent to even getting involved in the project in the first place.

Make the Issue Tracker, Wiki and Hipchat Channels Public

One key turnoff for would-be Open edX contributors is the lack of a public issue tracker. While it’s understandable that edX would want to keep sensitive information out of public eyes, there really is no excuse for running an open source project without a public issue tracker.

Not does it convey an attitude of secrecy, but it causes duplication of effort because outside contributors are submitting bug reports for issues that have already been reported in the internal issue tracker.

Guarding this information behind a login screen is sending a message to the open source community - “You’re on your own to figure out that bug, because our bug tracker is not accessible to you.”

Were edX to open the issue tracker, it would potentially welcome many more contributors who could track down these bugs and fix them. But without knowing what bugs are outstanding, a contributor is left to finding these bugs on their own, bugs which may not necessarily be the highest priority.

OpenHatch is an organization that provides opportunities for people to build new skills by getting involved with open source projects and contributing to them. The OpenHatch site crawls these open source projects’ bug trackers and sends students on “missions” to fix these bugs.

edX is missing out on attracting these kinds of community activities because the bug tracker is not publicly accessible.

For every month that goes by, there are more bugs accumulating which will only make it harder to do later. More importantly, every month there are developers evaluating Open edX as a potential platform. Who knows how many run away when they discover there is no public issue tracker and deem it not a mature open source project.

This presumes that edX has fixed the process for integrating pull requests - we don't want people fixing bugs only to have the pull requests molder, or worse, ignored because someone inside edX is working on same bug.

I recommend to assign someone at edX to go through the existing private JIRA issue tracker and remove all sensitive information so that the tracker can be made public and accessible to all.

Similarly, the Confluence Wiki should be made public, or else move all relevant content to the Github wiki. The Hipchat channels should be made public, or else move all conversations to the public IRC channel.

Other recommendations

Here are some other recommendations for community building activities:

Hire a full-time community manager whose only job is to service the Open edX community. This person might work on the following things:

• Invite people from Open edX community to regular Hangouts to talk about what they’re working on. Make it welcoming to newcomers who are curious. Invite and feature guests.

• Create an Open edX Google+ community to share links and other resources. See OpenShift as a good example: https://plus.google.com/communities/114361859072744017486 

• Open edX newsletter (subscribe to Google alert for “Open edX”)

• see Docker.io newsletter

• Call for sub-communities to form around installers, translations, learning analytics, XBlocks, etc. Self-organizing but with a clear sub-community leader who’s responsible for scheduling and documenting meetings. See Jono Bacon’s Art of Community for recommendations.

• ex: http://plone.org/community/teams

• Organize Open edX meetups - for users and developers (not just students).

• ex: https://www.docker.io/meetups/
• ex: http://plone.org/support/user-groups
• encourage people to step up and organize the meetup: https://www.docker.io/meetups/organize/

• Open edX tuneups - weekly online meetings to enlist community members to squash bugs.

• ex: Plone tuneups: http://plone.org/events/plone-tuneups

• Register Open edX on OpenHatch (http://openhatch.org/) a free service that matches contributors with open source projects

• “Superstar” award to most helpful person in IRC channel

• ex: http://plone.org/foundation/2009-help-channel-superstar

• Organize book sprint to bring together experts from the community to produce a book in 3-5 days.

• I know Adam Hyde, the founder of www.booksprints.net and www.flossmanuals.net and he connected me with...
• Anne who organized the OpenStack book sprint. Here is her proposal: https://wiki.openstack.org/wiki/Documentation/OperatorsBookSprintProposal
• Hear from book sprint participants: ex: http://justwriteclick.com/2011/10/21/google-summer-of-code-doc-summit-stories/
• OpenStack book sprint created an O’Reilly book in 5 days
• http://opensource.com/business/14/4/oreilly-openstack-operations-guide

• See the finished product here: http://docs.openstack.org/ops/

• Organize a Open edX 1 year birthday event for the wider community of open source contributors

• ex: http://www.meetup.com/Docker-meetups/events/166783242/

• Organize a strategic planning summit to bring together stakeholders and align objectives

• ex: http://plone.org/events/2008-summit

Establish Clear Governance and Provide Ways for Collaborative Decision-Making

There are many governance models that an open source project can adopt. Right now, it’s not clear what governance model Open edX is using, and this creates unnecessary confusion.

In the conversations that I had, there was discontent with the way outside contributions are handled. It’s paramount to make very clear guidelines for contributors, so they know what to expect.

In regards to the public roadmap, many expressed concern that the roadmap seemed to be an announcement of what edX was planning to build, not a dialog about what features are needed by other organizations using Open edX.

There needs to be a way for institutions who’ve adopted the Open edX platform to make their voices heard, and have some influence in the direction the software is going. While they may not be contributing financially, they are contributing, in the case of Stanford, with significant developer resources and new feature development.

There seems to be a lack of transparency about how decisions are made, and this creates an environment in which outside contributors feel left out of the process.

Here are my recommendations:

• Choose a governance model and communicate this clearly to the community.

• ex: Taverna Governance Model. http://www.taverna.org.uk/about/legal-stuff/taverna-governance-model/
• ex: Ubuntu Governance Model. http://www.ubuntu.com/about/about-ubuntu/governance
• ex: Docker Governance Advisory Board: Draft Proposal Version
  https://www.docker.io/governance/
  why an advisory board? http://blog.docker.io/2014/04/docker-open-governance-advisory-board/

• Establish clear guidelines for how pull requests are reviewed (perhaps delegate more responsibility to competent community members)
• Establish a community council comprised of contributors who can be the voice of the community and help shape the future of the Open edX project.
• Establish a framework team or technical advisory committee who are competent enough to review new features proposed by community members and deem whether they will be included in the next release or not. Make this process transparent so contributors can see why a feature was or was not accepted.

Evangelism and Outreach

Open source projects are no different than commercial products in that they need to be marketed, promoted and evangelized. A concerted effort needs to be made to reach beyond the inner circle of contributors to spread the news about the software. This means giving talks at local meetup groups and conferences. These talks don’t necessarily need to be given by hired edX staff. In fact, it’s almost better if they are given by members of the community. But this means actively recruiting those persons who are using Open edX to give such talks.

Here are some other recommended marketing and outreach activities:

• Recruit members from community to form “Open edX marketing/evangelism committee”
• Allocate budget to sponsor and speak at popular open source conferences.  

• OSCON -- http://www.oscon.com/
• FOSDEM -- https://fosdem.org
• Open Source Bridge -- http://opensourcebridge.org/,
• PyCon -- http://us.pycon.org
• DjangoCon US -- http://www.djangocon.us/ and DjangoCon EU -- http://djangocon.eu
• EuroPython -- http://europython.eu
• SXSW Edu -- http://sxswedu.com/

• Produce 1 page marketing brochure to hand out at events (Creative Commons licensed and include source docs to allow for modifications and translations)
• Allocate budget for schwag - t-shirts, stickers, etc
• Participate in Google Summer of Code  (GSoC) and find mentors who can help students during summer.  https://developers.google.com/open-source/soc/