edX Architecture Onboarding��Last updated: Feb 2019

edX Architecture

Part 1: High-level Overview

Part 2: Data and Scale

Part 3: Technical Feature Detail

Part 4: Technical Development

​

​

Part 1

edX Architecture: High-level Overview

• What is edX?
• Where are we today?
• Where are we going?

​

​

Part 1

edX Architecture: High-level Overview

• What is edX?
• Where are we today?
• Where are we going?

​

​

edX Mission

To create the world’s leading learning destination site and platform so that everyone around the world can become better educated.

We will do this through:

​

increasing access to high quality education,

improving quality of education

on-campus and on-line,

and enabling research.

edX Themes

• D
• R
• E
• A
• M
• C
• O

• iscovery
• evenue
• nterprise
• rchitecture
• asters
• ontent
• perational Excellence

edX Core Value

• Scalable Quality Education with Support
• Partner Brand recognition and amplification
• Connecting People, Learning Content and Corporations
• Content Stackability with Credentialing
• Flexible Platform for deep learning, experimentation and pedagogy
• Inclusivity: open to all learners, affordable, open source

Part 1

edX Architecture: High-level Overview

• What is edX?
• Where are we today?
• Where are we going?

​

​

Data Flow

Authentication

Authorization

Authorization

Part 1

edX Architecture: High-level Overview

• What is edX?
• Where are we today?
• Where are we going?

​

​

https://radar.thoughtworks.com/?sheetId=https%3A%2F%2Fraw.githubusercontent.com%2Fedx%2Fopen-edx-proposals%2F349c76072ff302a999e2bfc159e902d527f8cca4%2Ftech-radar.csv

Part 2

edX Architecture: Data and Scale

• Content
• Analytics
• Scale

​

​

Part 2

edX Architecture: Data and Scale

• Content
• Analytics
• Scale

​

​

Content

User

Data Sources of Truth

Course Structure

XBlock: reusable building block for course content��Course structure: directed acyclic graph of XBlocks�

XBlock types:

• Structural (platform-provided)
• Section = chapter
• Subsection = sequential
• Unit = vertical
• Component (platform-provided + custom-built)
• problem (Checkboxes, Dropdown, etc)
• html (Text, Announcement, etc)
• video
• discussion
• openassessment (Open Response Assessment)
• drag-and-drop-v2
• library_content (Randomized Content Block)
• Orphan (not in course hierarchy)
• course_info: handouts, updates

LMS View

Course Outline

Course

Section

Subsection

LMS View

Courseware

Subsection

Section

Unit

Unit

html�component

CAPA problem�component

Content Size�(# of XBlocks in edx.org courses, with 60 types of XBlocks)

XBlock Types on edx.org

annotatable

chapter

combinedopenended

conditional

customtag

discussion

done

drag-and-drop-v2

edx_sga

edx-reverification-block

error

google-calendar

google-document

graphical_slider_tool

html

HTML

imageannotation

library_content

lti

lti_consumer

​

problem

problem-builder

problemset

randomize

rate

recommender

schoolyourself_lesson

schoolyourself_review

sequential

split_test

step-builder

survey

textannotation

ubcpi

vertical

video

videoalpha

videosequence

word_cloud

wrapper

​

mentoring

officemix

openassessment

oppia

p

pb-answer

pb-answer-recap

pb-choice

pb-column

pb-completion

pb-dashboard

pb-instructor-tool

pb-mcq

pb-message

pb-mrq

pb-table

pb-tip

peergrading

poll

poll_question

​

XBlock views provide HTML and JS to run in user’s browser.

XBlock developers implement front-end and back-end code,

requiring approval to install on edx.org.

Backend code must be written in Python.

XBlock backends run in an edX platform (LMS or Studio) runtime. The runtime provides out-of-the-box services such as internationalization, etc.

The Field Data runtime service provides a data storage abstraction for user data (in SQL) and content data (in mongoDB).

The modulestore is edX’s abstraction for content data.

Course Content Extensions

• Installation required on edX server
• XBlock
• custom frontend (HTML/JS) and backend (Python)
• Installation not required
• HTML XBlock
• custom frontend only
• (currently) not sandboxed
• JSInput Problem
• custom frontend view
• custom grading (Python)
• LTI
• need external server to host content
• frontend view constrained within an iFrame
• External Grader

Part 2

edX Architecture: Data and Scale

• Content
• Analytics
• Scale

​

​

Analytics

Analytics

Part 2

edX Architecture: Data and Scale

• Content
• Analytics
• Scale

​

​

edx.org Usage Growth�(per quarter)

edx.org Usage Growth

(compounded)

edx.org Content Growth�(# of courses)

Developer Growth

(edX employees only)

Web Usage

All

Desktop

Mobile & Tablet

1 Week

1 Month

1 Day

Tues

Web Usage

(courses.edx.org only)

All

Desktop

Mobile & Tablet

1 Week

1 Month

1 Day

Mobile Usage

All

Android

iOS

Web Performance�(average across all LMS pages)

requests per min

avg response time

Web Performance�(LMS Courseware page)

requests per min

avg response time

percentiles of response time

Scaling with Reactive Manifesto

Elastic

LMS: 13k rpm�Studio: 60 rpm

no bottlenecks�

Responsive

99% < 2s

consistent

Resilient�99.9% uptime

MTTF & MTTR

isolation

Scaling with Reactive Manifesto

Elastic

LMS: 13k rpm�Studio: 60 rpm

no bottlenecks�

Responsive

99% < 2s

consistent

Resilient�99.9% uptime

MTTF & MTTR

isolation

Message Driven

event-driven

asynchronous

loosely coupled

location transparency

Scaling in devOps

• Environments have own AWS Virtual Private Cloud
• prod, stage
• Deployments have own AWS account, secure-config repos
• edx, edge
• Clusters are independently deployed applications (IDAs)
• edxapp-cms, edxapp-lms, discovery, xqueue, insights, forums, ecommerce, analytics-api, credentials
• Create triplets: prod-edx-discovery

Scaling in devOps

• Auto scaling groups (ASG) for edxapp
• Production: typically 7-12
• Edge: 2-4
• Stage: 2
• Each ASG typically has 22 instances
• MongoDB
• Primary & secondary - both are readable
• CMS - accesses primary only
• LMS - accesses primary or secondary
• MySQL
• Replica failover provided by RDS
• Secondaries spread across Availability Zones

​

Part 3

edX Architecture: Technical

Learning Tools Interoperability (LTI)

• Connects learning systems (LMS) with external tools.
• Standard created by IMS Global Learning Consortium
• Version 1.0 (2010)
• LTI Tool Consumer
• Typically an LMS that consumes external tool
• Provides user and context to the Provider.
• LTI Tool Provider
• Provides learning service/tool to the Consumer.
• Can be hosted anywhere.
• Version 1.1 (2012)
• Introduces LIS Outcomes Service for Provider to pass back scores to Consumer.
• Version 2.0 (2014)
• Introduces Tool Proxy and REST-based communication between services.

LTI 1.1 Config

• Tool Provider administrator provides configuration and credentials to Tool Consumer administrator as an initial out of band exchange.�
• For Open edX, configuration information is entered in Course and LTI consumer block settings as described in the documentation.�

LTI 1.1 Launch

Whenever a user accesses an LTI link, the Tool Consumer “launches” the LTI tool in an embedded iFrame or a new browser window by sending a POST request to the Tool Provider.�

Fields sent in HTML Form to TP

* context_id is identifier of TC's context� Context may be of type:

1. Course template

2. Course offering

3. Course section

4. Group

5. None if launched outside of context�

* resource_link_id: unique per TC's usage of the LTI link�

* user_id: non-identifiable, primary-key of user�

* custom parameters requested by TP in its configuration

​

If scores are passed back to TC:� * lis_outcome_service_url: where to pass back� * lis_result_sourcedid: unique for (context_id, resource_link_id, user_id)

​

LTI 1.1 Score API

• A Tool Provider may optionally send scores back to the Tool Consumer.�
• As server-to-server calls, these score management APIs may be called at any time.�
• lis_result_sourcedid remains as the unique identifier for the (context_id, resource_link_id, user_id) tuple.

​

Automated Communication Engine

https://github.com/edx/edx-ace

Schedules

Command Query Responsibility Segregation (CQRS)

• Writes
• Course Team Content edits
• Learner Enrollment changes
• Reads
• Asynchronous schedules task management

​

Separation of Concerns

• Schedules
• Higher-level business and policy layer
• Enrollments, dates and deadlines
• Automated Communication Engine (ACE)
• Lower-level detail layer
• Pluggable Notifications framework

​

Data-driven development

• Monitoring of errors and metrics via�New Relic and Splunk Logs
• Data analysis via Tracking Logs and GA

​

​

Schedules README

XQueue

XQWatcher Pull Model

• “Active” model that constantly polls for submissions.�
• Recommended pattern for external graders�

XServer�Push Model

• “Passive” model that registers a web interface to be called by XQueue.

​

• Deprecated pattern in favor of other xBlock integrations such as LTI.

eCommerce

• Each user has only one Basket.
• edX.org sells seats, not content.
• Code uses Django Oscar and its models.
• Otto is the pet name for our edX extensions.
• Product classes
• Entitlement: course id, seat type
• Course Run Seat: course run id, type, verification required, credit provider, num of credits
• Coupon: B2B
• Enrollment codes: quantity of seats with a CSV of codes to enter in a Basket.

Purchase Flow

• Order fulfillment happens asynchronously from the checkout workflow.
• User is taken to the receipt page before order is fulfilled.�
• ECOM, ECOM Worker, and LMS are each separate IDAs. The enrollment path depends on each of them to be responsive.
• ECOM Worker is a thin wrapper above Django celery in the context of order fulfillment, calling REST APIs in the ECOM process - unclear of its value in the fulfillment flow.
• Order status remains open until enrollment is successful.

Forums

Django Frontend

TODO

Topics/Features that still need slides

• SSO & OAuth
• JSInput
• CAPA

Part 4

edX Architecture: Technical Development

edx.org Server Environments

(running on AWS services, except Devstack)

• Production:
• Prod: Hosts courses published on edx.org marketing site
• Edge: Secondary production environment with few additional features enabled; courses not published on marketing site
• Stage: Final testing environment before deployment to Prod
• Sandbox: Temporary servers created by developers for testing
• Load: Shared resource for load testing
• Devstack: Localhost development and testing

Tools & Technology (1)

• Languages:
• Server-side: Python 2.7
• Client-side: ES6/ES2015, HTML, CSS (Sass)�deprecated: Coffeescript, ES5
• Web frameworks:
• Server-side: Django 1.8.13 -> Django 1.11
• Client-side: React, Redux, Bootstrap, Webpack�deprecated: Backbone, Underscore, RequireJS
• Storage: MySQL, MongoDB, S3
• Search: ElasticSearch
• Docs: ReStructured Text, Sphinx

Tools & Technology (2)

• Analytics
• Processing Environment: Hadoop (EC2 & EMR), Luigi
• Storage: HDFS (AWS S3) + Hive, MySQL, Vertica (Postgres DB) on EC2
• Visualization: Google Analytics, Tableau, DataDog
• Mobile
• OS and Languages: Android (Java) & iOS (Swift, ObjectiveC)
• IDEs: Android DevStudio, Xcode
• Build tools: Gradle

​

​

Monitoring & Data Analysis

• New Relic – performance and crash analysis
• Splunk – log analysis
• Datadog – statistical analysis
• Crashlytics (Fabric) – crash analysis of mobile apps
• Coursegraph - graph query of course content
• Google Analytics – usage analysis
• Analytics Pipeline (Vertica and Tableau)
• Local Profiling: PyInstrument, RunSnakeRun

Coding Standards

• Internationalization (i18n) / Localization (l10n)
• RTL (Right-to-Left)
• Accessibility (a11y)
• REST API Conventions
• Quality linters: PyLint, ESLint, etc
• Test Coverage
• Security
• Clean Code

Testing

• CI: Jenkins, Travis
• Unit tests: Python unit tests
• UI Unit tests: Jest (JS Tests)

deprecated: Karma/Jasmine

• Acceptance tests: Bokchoy�deprecated: Lettuce
• Load tests:
• Server-side: Locust.io
• Client-side: sitespeed.io
• Task management: paver, npm scripts

Security

• Goals
• Confidentiality (who can read)
• Integrity (who can edit)
• Availability (prevent denial of service)
• Authentication
• Django Authentication classes
• Session cookies (Web transactions)
• OAuth bearer tokens (API transactions)
• JWT tokens (server-server transactions for now)�deprecated: API Keys
• Single Sign On
• Authorization
• Django Permission classes
• LMS: has_access
• Server-side checks; can’t trust client

Security Attacks

• XSS (Cross-site scripting)
• Safe Templates Linter
• CSRF (Cross-site request forgery)
• Safe Endpoints Middleware
• CSRF Middleware
• exceptions: CORS (Cross-origin Resource Sharing)
• Safe Sessions Middleware
• Binds user to session cookie
• Enforces cookie expiration

​

API Design

• Authentication: OAuth2/JWT access tokens using Bearer authentication scheme.
• Authorization: Server-side access control checks for all data access.
• Client/server separation of concerns
• Business logic centralized on server
• Pluses: Thinner clients and future maintainability
• Minuses: Offline support is difficult
• Performance:
• Response Time: 95% response time < 2 seconds
• Round Trips minimized. Tradeoffs between payload size, pre-fetching, and client-side caching of the data.
• Payload Size minimized by parameterized APIs (filters, etc).
• Strategy: Favoring Generic APIs over Mobile-specific ones
• API Conventions: Wiki document

​

Part 5

edX Architecture: Sustainability

Developer Documentation