A journey through migration of a monolith to a multi tenant cloud platform and lessons learned.
About me...
Graeme Baillie
Cloud Solution Architect & Community evangelist
SWITCH ( NREN Switzerland )
Target Audience
IT Administration
Cloud Administration / Engineering
Migration Projects
What is our product?
On-Prem Video Encoding and Learning Management System.
What is our dev/ops environment?
Target?
Our Reaction??
What next?
Everyone else?
Our Reaction??
The Bombshell
Migrate the container platform!
Reaction?
Huge questions!!
Lessons learned
Not clear to anyone at decision point what the scope and size of such a migration to the most critical component - Think carefully and plan it !
Little or no understanding of dependencies e.g. ci/cd, monitoring systems which may have to be re-done - Map out the existing platform, dependencies and future deficiencies before any decision!
Large re-architecting required, not just Kubernetes->Kubernetes - Understand how two platforms differ and plan accordingly
Migrate the ci-cd tools!
Reaction?
More ( huge ) questions
Lessons learned
Migrate ci-cd at the same time as a platform migration - one more unnecessary variable - don’t do it
Ansible playbooks had to be re-written - we were not sure exactly what the dependencies were
Was found that it was not possible to deploy natively containers - had to mis-use multiple CodeBuild projects - not a good idea
We still had to continue Product development with existing ci-cd - too difficult to plan migration
Split into two tracks
PODS ( SCRUM )
Cloud Architecture/Monitoring/Platform
Product Development/ci-cd
Application Migration Scope
Lessons learned
Don’t aim for perfection with MVP 1.0
Don’t keep moving the goalposts - Clearly define the scope and stick to it
Realise no system is EVER finished - aim for constant evolution, continual improvement
Understand the available resources
Plan for the unexpected - Don’t book every second till the project end
Application Design - Front End ( simplified )
http://tenant1.app.company.com/browser
http://tenant1.app.company.com/editor
http://tenant2.app.company.com/browser
http://tenant2.app.company.com/editor
http://tenant1.app.company.com/browser
-> Service “tenant1/browser”
http://tenant2.app.company.com/editor
-> Service “tenant2/editor”
Application Design - Content Distribution
Single CloudFront distribution with 2 origin behaviors.
Signed URLs provide restrictions to content as well to add an extra layer of security.
Single S3 bucket with separate folders for each tenant, and subfolders matching the caching patterns
Lambda@Edge to rewrite URL’s for particular tenants to remove this overhead from the application
Files only available through Cloudfront and not direct with s3 due to OAI ( Origin Access Identity )
Application Design - Distribution workflow
1. Request comes into Route 53 requesting content for the first time. Request passed from CloudFront to Application http://tenant1.app.company.com
2. Application creates signed URL with get request to media folder in S3 bucket where content resides.
Lambda @ Edge catches the viewer, request and rewrites URL to add the tenant name to the path.
https://tenant1.app.company.com/tenant1/video/video1.mpg
3. Cloudfront forwards origin request to specific subfolder in S3 bucket.
4. Data is retrieved to end user for consumption and cached by Cloudfront if it matches any cache behaviors.
5. User makes same request for video1 later. Request is rewritten by Lambda and the request stops at CloudFront cache and finds a matched cache object.
6. User consumes cached object directly
Lessons learned
Lambda@EDGE an early case of over-engineering - don’t waste time on efficiencies that are clearly no longer required!
AWS as a Platform - Scope
A Platform for growth
security
Logging
auth
shared
monitoring
tools
App-1 prod
App-1 Staging
App-2 prod
App-2 Staging
Playground-1
Playground-2
federation?
GLOBAL
APP SHARED
APPS
R&D / TESTING
Lessons learned
Global SSO/Authentication system
Avoid Physical IAM User hell with custom inline policies
Centralise logging/monitoring/ci-cd in own accounts and design so they can be used by multiple teams/projects/apps
Security account for audit/expedited access
Centralised secret management ( secrets manger / parameter store )
Lessons learned - there are more
MAKE IT EASY FOR EMPLOYEES TO USE THE PLATFORM!
Allow to connect to accounts with minimum of fuss
Enforce MFA & Password policies
Single place to login, then assume roles/Short term creds
Design & Promote tooling ( aws-cli, profiles, aws-vault )
Tooling for edge cases ( e.g. EKS )
Sustainability
Should be planned from day 1
Beware consultants - priorities may be different
Insist on Operations / Patching / Support concept be built before rather than at the end.
Decide on Monitoring / Logging stack early - ensure developers take this into consideration every day.
Insist on Auto-Scaling / Elasticity as much as possible.
Sustainability (2)
Use Infrastructure as Code tools - FROM DAY 1 - Configuration Management will save the day
Top 5 Takeaways