1 of 19

Rootconf 2017 // Ruchi Singh

Migration of 300 microservices from AWS Cloud to Snapdeal Cloud

@rchsngh569

2 of 19

The next 15 mins...

Who are we?
Why did we migrate 300 microservices
Planning for migration
Runtime gotchas and our fix to them
Rollback strategies in case of failure
Key learnings
Party time

3 of 19

Largest e-commerce marketplace in India

Buyer Seller

4 of 19

Overview Of Snapdeal Cloud:Cirrus

5 of 19

And the story begins…..Our plan for migration

6 of 19

Checklist before starting migration

Understand why you’re migrating
Map out a clear plan
Regaining knowledge
Risk reduction & ensuring compatibility
Network particularities & limiting latencies
Get everyone onboard

Before starting anything, we always make a checklist. So did we. so we figured out what made us to start such a big project. We listed down the main factors/reasons first, Where we were lagging behind, or If we see from a business point of view, were we spending a lot for our infra need, performance issues, security and many more. After that, strong planning was needed for this project. This includes all such considerations --

---Regaining knowledge - gaining knowledge about infra was the main task for us. Some applications may be decade old. So we decided to have a complete knowledge about our existing environment. How many services are currently running.

Extended or unplanned downtime were the main risks that we had to recognise and try to mitigate them. To what extent our business can tolerate these risks will depend on the importance of the application that we were migrating at a particular point of time.

Similar calculations needed to apply on risk of data loss. The more important sensitive data, the more safeguards needed to be in place to prevent its loss during migration. So we thought of some disaster recovery process and extensive backups that continued to work during migration. Then for all applications , we thought of a trial of complete migration on our staging environment to ensure the compatibility of new data center and the applications we have. Are there any code changes required in applications.

--- network particularity --- in terms of network, we had to ensure that each service should have a pre-determined space in new network. And for this, it is important to consider all the aspects of firewall settings, domains and trust certificates to ensure full compatibility

---limiting latency--- controlling latency was the most important concern for us. Specially, for those services which are critical for business. Knowing exactly which services work together and the frequency of their communication can work in terms of controlling latency.

And of course getting everyone onboard for such a large project was also a challenge for us.

7 of 19

Why did we build a private cloud?

8 of 19

Cost - A major driving factor

Public clouds are great till the growth is unpredictable

At an inflection point, public clouds don’t remain cost effective, we needed an alternate.

And how we made it cost effective -

SD cloud is built using 100% open-source components.

Some analysis on some enterprise technologies and calculation of operational cost gave us clearness about our idea.

We have a team to build, automate and manage the DC and Cloud Platform.

So cost was one of the major factor. Like I said before that we were in hyper growth mode and as our business grow, our infrastructure requirements also continue to grow and our bill on the public cloud was phenomenal. So we had to find ways to reduce that bill or to control that one. It was very clear to us that at a certain scale, public clouds stop being cost effective. They are okay when you are just starting up but at an inflection point, you need to start looking for some alternatives and for us it was building our own cloud.

Snapdeal cloud is completely built on open source technologies. We did some analysis on some enterprise technologies. We also calculated the operational cost of a managing a data center and we came up with this plan. We have a small team to build, automate and manage the DC platform.

9 of 19

Other factors...

Performance - One machine one service; much higher performance, optimized it for self use

Security - Advance Enterprise firewall, Intrusion Detection, DDOS Prevention

Data sovereignty - keeping our critical data within boundaries

The next big reason for us to do so besides cost was performance and security. We wanted to get more performance from our infra and in public cloud you are restricted because you are in a shared tenant architecture and there is only so much performance that public cloud offer. They can offer you more but with a very high cost but it is restrictive for the large scale that we are looking for. So by building our own cloud, we are now able to optimize our infra for self use.

We are able to put advance security appliances , D-dos prevention, intrusion detection into our data center so definitely a step-up into the security that public clouds offer.

And lastly, data sovereignty and compliance was also a big reason for us. We have a digital wallet application which required us to make sure that all the money related data we store in that wallet remains within the boundaries of India and at that time, public cloud provider we were using did not have a region in India.hence we had to look for, at least for that particular application, to be hosted within the boundaries of India. .

10 of 19

To Summarize...

11 of 19

Gaining knowledge about infrastructure...

12 of 19

After planning steps

Listed down all the services and their dependencies in a central place which is yaml file act as a infrastructure as code
Divided services in smaller groups, kept all those together which were tightly dependent
Made a dependency graph to facilitate the migration by showing dependency between services
Kept all datastores in replication mode and ensured that data is in sync continuously
Migrated tightly coupled services together according to the business flows

13 of 19

Gotchas during migration and our fix to them

-- Security Groups to redirect traffic between services running in old cloud and new cloud

-- Data not in sync issue so took a delta and dump that data to the new machines

-- launched machines were not able to handle the load so extend our infra at run time.

-- Strong monitoring needed for new system and applications was required and we were missing some parts in monitoring. We use EFK, Icinga for monitoring

Planning is most important key for every project but there are always some gotchas which you get only at runtime.

And we also did some mistakes during the migration of some critical applications.

There are some scenarios, one of them was when we were in the middle of our migration, we used to put some security groups in our old cloud on the ELB to serve the traffic to the actual servers and there were some services which used to communicate frequently and one of them was running in the the public cloud, and when we were migrating the 1st service so traffic was supposed to go on public cloud according to our code or set up but we forgot to place the security groups there and there was no traffic coming on the public cloud servers where the actual service was running. So we had to check all these check before serving the actual traffic in production environment.

Second one was we used to keep our databases in sync so that no data corrupt things happens but there were some scenarios where our data had been corrupted and we were getting serious errors in our applications. So in that case we took the delta and dump to the new database servers. But that was kind a pain for doing so. So we had to make sure that our database servers are in sync correctly so that no data loss happens.

Another scenario was when we migrated one of our service, the launched machines were not able to handle the load so we had to extend our infrastructure at run time. So to avoid these risks we had to ensure the number of machines needed for that application to handle the load.

And to avoid all these issues, monitoring of your infrastructure is one of the most important thing. We were doing monitoring of every individual system as well as our applications. In our case, order count was the critical thing and we had to ensure that our order count doesn’t drop during migrations activities. We use Icinga and EFK stack for our monitoring purposes.

14 of 19

Our plan for failure...Rollback strategies

If a service migration fails for any reason, some rollback strategies we have -

Traffic redirection - switching the pointing of applications
Database resync - database servers in old cloud and in new cloud are in sync for few days and If we see any failure, we just change the pointing of application to the old database server
Cooling period - we set a cooling period to shut down the servers in old cloud

As we did a strong planning and trial migration on our staging environment but before doing that on production, we set some rollback strategies in case of failure.

So if a service migration fails for any reason, there are some of the rollback strategies e set -

The bigger one was how to redirect the traffic if things don’t work out in new cloud. So we thought to have a switching type service so that within miliseconds we could redirect the traffic to the application running in old cloud. We were using some service discovery code we wrote on our own. There is a service called nerve which we written on our own using dendrite (a service discovery tool) which is installed on every server in new cloud and we are keeping the IPs of old servers in yaml files for each application. That service read these info from yaml file and based on that redirects the traffic to corresponding servers. So that’s how we were redirecting the traffic to the old servers in case of failure on new data center.

Database resync - before redirecting the traffic, database resync was also one of the thing we thought of doing so that application could point to the updated database server. So how we were handling this so initially when we used to deploy a service on the old cloud, we used to keep the new database servers in read mode and only allow read queries to be execute until qa person doesn’t give us the green signal. As long as we got the green signal, we changed our database servers in the write mode to allow the actual traffic to be served.

Cooling period - we set a cooling period to shut down our old servers. That was a around a month until we find that service is running flawlessly in the new data center.

15 of 19

Execution part : technical tools

-- Our yaml files for each individual service (our infrastructure as code source)

-- Dendrite for service discovery (nerve and synapse)

-- Saltstack, chef (orchestration tools)

-- Git, Jenkins (CI/CD pipeline)

-- Automation scripts

16 of 19

Key Learnings

-- Plan, Plan and Plan for your cloud migration! This is where your project fails.

-- Understand your services, architecture and their dependencies

-- We created a live service dependency graph to facilitate migration

-- Don’t migrate as-it-is, fix the problems that you never got time to fix

-- Strict naming conventions and make sure all launched services are registered with all orchestration tools you are using

-- Automate and monitor everything!

17 of 19

But our cloud is hybrid...

But we didn’t stop there, still using public cloud for these purposes ---

-- Disaster Recovery : for data backup and recovery

-- New service/company acquisition: new company acquires which used to run in public cloud and takes sometime to migrate it

-- On-demand : at critical times like diwali. Reach maximum capacity and still need to grow

But we didn’t stop here. Our private cloud is 100% hybrid. We are still using some part of public cloud for some reasons such as disaster recovery, new company acquisition and on-demand scalability at critical times. we still keep our data in the public cloud storage as backup. Second reason is if we acquire a new company in future which used to run in a public cloud so it takes sometime to migrate its whole infra to our private cloud. So for the time being, we can run it’s infra in the public cloud it is using. And Last but not the least, on-demand facility. Which is like, at critical times like diwali and some festive seasons, we reach to the maximum capacity and the demand is to still to grow and it takes time to expand our infra. At that point of time we can use public cloud infra to use it at runtime so that we could run our business flawlessly.

18 of 19

And it’s party time!

After 1.5 years, ups and downs, few downtimes, challenges and Snapdeal.com is running on it’s own cloud.

19 of 19

Thank you!