1 of 61

The Wild West of High Availability in Open edX

2 of 61

Copyright © 2017 OpenCraft GmbH

This text and images in this presentation are released under the Creative Commons Attribution-ShareAlike 3.0 licence, except for any logos.

Code samples are released under the AGPL v3 license unless otherwise noted.

Brandon DeRosier

brandon@opencraft.com

Feanil Patel

feanil@edx.org

3 of 61

Agenda

Managing Infrastructure

Fork Management

Choosing Your Stack

Examples

4 of 61

Choosing your stack

5 of 61

Outline

  • Infrastructure Choices
    • On Prem
    • AWS
    • OpenStack
    • Other Cloud Providers

6 of 61

On Premise

  • Most Control
  • Most expertise required
    • You'll be setting up a lot of it yourself
  • You might need this based on your organization or local law requirements

7 of 61

Amazon Web Services

  • Most managed services Available
  • Used by edX Inc.
    • Our tools can become your tools
    • Con: Our tools are not always built for everyone

8 of 61

OpenStack

  • Pro: Used by OpenCraft
    • OpenCraft adding support into existing edX tools where possible
  • Con: no two openstack clouds are the same

9 of 61

Other Infra Providers

  • Usually provide the basic abstractions you need
    • Compute
    • Load Balancers
    • Storage
    • RDBMS as a Services
  • Biggest differentiators are usually price, reliability, and tooling availability

10 of 61

Example:

The bare minimum

11 of 61

Sandbox

  • Not Stateless
  • Not Scalable
  • Not Highly Available

12 of 61

Sandbox?

… sort of

  • Not Stateless
  • Not Scalable… kind of
  • Not Highly Available��
  • Can fully monitor services separately
  • But, more single points of failure

13 of 61

Definitely not

a sandbox!

  • Not Stateless
  • Not Scalable
  • Not Highly Available��
  • Can fully monitor services separately
  • But, more single points of failure
  • No single points of failure

14 of 61

If desired, use managed databases

  • Less infrastructure to take care of
  • Easy to scale�
  • More expensive than the bare infrastructure�
  • Check for education/user privacy law compliance

15 of 61

16 of 61

Example:

A large stack

17 of 61

Where we left off:

18 of 61

Separate forums...

​

​

19 of 61

Separate workers...

​

​

20 of 61

Analytics!

​

​

21 of 61

€commer¢e $$$

​

​

22 of 61

Well...

​

​

How do we make this smaller?

23 of 61

Well...

​

​

How do we make this smaller?

Maybe try the managed services again…?

24 of 61

External RabbitMQ...

​

​

​

​

25 of 61

External MongoDB...

​

​

26 of 61

Hmm...

​

​

That didn’t do very much….

27 of 61

Hmm...

​

​

That didn’t do very much….

But maybe we can replace all these load balancers…?

28 of 61

Service Discovery!

​

​

29 of 61

Service Discovery!

​

​

Very little benefit for a small

deployment, though...

30 of 61

Managing Infrastructure

31 of 61

Tools

  • For things that change slowly
    • Ansible Infrastructure Modules
    • Cloudformation
    • Terraform

32 of 61

Tools

  • For things that change more often
    • Configuration Repo
    • Asgard
    • GoCD

33 of 61

34 of 61

35 of 61

36 of 61

37 of 61

38 of 61

39 of 61

40 of 61

41 of 61

42 of 61

43 of 61

44 of 61

45 of 61

46 of 61

Fork management tips

47 of 61

Fork management tips

The Four Horsemen Of

Technical Debt

48 of 61

1. KEEP YOUR DIFFS SMALL

  • Tiny stuff adds up quickly
  • Investing upfront pays off here - deal with it before it’s a problem
  • Named release rebasing will start costing you a fortune otherwise
  • Huge diffs are simply a nightmare

​

49 of 61

2. Upstream Everything

  • Distribute the burden of Maintenance
    • Someone broke your feature’s tests?
    • They have to fix it before merging
  • Build better software
    • Forces your architecture and code to be high quality
  • Be a good citizen

​

50 of 61

3. Don’t edit Ansible output

  • lms.env.json
  • lms.auth.json
  • cms.env.json
  • cms.auth.json

​

  • Do it like edX: Use configuration management
  • These files may not always be the config destination
  • edX configuration keeps up as the schema changes
  • Things are actually much simpler this way

​

51 of 61

3. Don’t edit Ansible output

3.5. Also, don’t edit settings code directly

  • lms.env.json
  • lms.auth.json
  • cms.env.json
  • cms.auth.json

​

  • Do it like edX: Use configuration management
  • These files may not always be the config destination
  • edX configuration keeps up as the schema changes
  • Things are actually much simpler this way

​

  • lms/envs/common.py
  • cms/envs/aws.py
  • ...etc.

​

52 of 61

4. Avoid template overrides in themes

  • Sometimes you have to do it
  • ...but try to make style modifications as much as possible
  • Context changes in views will have you fixing your templates every release

​

53 of 61

Brandon DeRosier

brandon@opencraft.com

Feanil Patel

feanil@edx.org

Questions?

54 of 61

Not part of presentation: Cloud Agnostic Stack Choices

  • The OpenEdx platform uses many off the shelf tools.
  • You may want to use SaaS providers
  • Rabbit as a service
  • Mongo as a service
  • Elasticsearch as a service

55 of 61

Infrastructure as Code Options

The next 3 pages talk about different tools for managing infrastructure as code.

56 of 61

Ansible Modules

  • Support
    • Number of Modules to manage ec2 resources is growing
    • Interfaces between modules not consistent
    • Modules not available for most other clouds
  • Imperative nature
    • Great for orchestration
    • Bad for predicting what’s going to happen
  • If you’re already using ansible, you don’t have to learn a new tool

57 of 61

Cloudformation

  • Declarative Syntax
    • Declare your resources in a file
    • Provide it to Cloudformation
    • Cloudformation ensures that all declared resources exist
  • Con: Only works with AWS
  • Con: Works best in one file
    • Can do multi files but is complicated
  • Pro: Supports Planning for making changes
    • You update your template and it can show you what it will do to your resources

58 of 61

Terraform

​

  • Similar in capability to current state of cloudformation
    • Declarative
    • Planning Capability
  • Differences
    • Can easily span multiple files
    • Can work with multiple cloud providers
      • Provider plugin system is very flexible

59 of 61

GoCD: https://gocd.io

  • Gomatic - DSL to make GoCD pipelines
  • Tubular - Convenience scripts and utilities
    • Can be used independently of GoCD
    • https://github.com/edx/tubular

60 of 61

Configuration Repo

  • Ansible playbooks for building App specific machines
  • Optimized for separate machines per application
    • Though roles can be combined flexibly
  • https://github.com/edx/configuration

61 of 61

Asgard

  • Used to managed deployments at edX
  • Allows for Imaged based blue/green deployments
  • Con: No longer supported by Netflix
  • https://github.com/edx/asgard

​