1 of 78

Transitioning to AWS in a hurry without getting owned (Hopefully...)

Devina Dhawan

02/06/2017 - Women & Non-Binary Focused intro to AWS

Email: 3@etsy.com

Twitter: @theulzo

1

2 of 78

Transitioning to AWS in a hurry without getting owned (Hopefully...)

Devina Dhawan

02/06/2017 - Women & Non-Binary Focused intro to AWS

Email: 3@etsy.com

Twitter: @theulzo

2

3 of 78

Introduction

3

Etsy (Jan 2015 - Present)
Orbitz (May 2014 - Dec 2015)
University of Illinois in Chicago

4 of 78

Etsy operates a global marketplace where people around the world connect,

both online and offline, to make, sell and buy unique goods.

4

5 of 78

Security at Etsy

Evangelizing Security at Etsy

Candy is a great way to make friends
Allow the conversation about security to be comfortable and inviting.

5

Security at Etsy is radical in that at its inception was already widely evangelized throughout the organization. Everywhere from product to systems engineering, security is major part of decisions that are made and it is even difficult to find a human that doesn't view security in a positive light.

Mostly because of a program we call security-candy, where we make candy available for the people who work at Etsy. This program exists to create a comfortable association with security and the people on the team. So, humans will come around to grab candy and may sometimes even have questions for us or will come by to talk to us about somehting they are working on. The relationships this candy builds is one low-cost method of evangelizing security.

Though, that is a gross over simplification for work that was done by the creators of the security team. So, as you can tell, Etsy is a magical place filled with unicorn engineers and is also a beautiful place to work.

AND WE ARE HIRING! We are hiring for managers, senior engineers and engineers for the Brooklyn office and for Remote work. I am a remote employee for Etsy and I love it.

6 of 78

What is this talk about?

I will help you improve your existing AWS infrastructure
You will walk away with action items
http://bit.ly/2EnZU1q

6

7 of 78

“Securing Amazon Web Services”

7

8 of 78

Traditional bare metal
Minimal footprint in the clouds

Infrastructure

8

9 of 78

Where to begin?

Previously I had mentioned how large the scope of this project was when it was handed to me, the reason is because at any growing company, you will have humans that will want to test things in various infrastructures. You go through acquisitions. You may need to run external scans of your infrastructure as well. And especially if you want contractors to build sites for you, you can’t just give them access to your entire infra. So in so many ways, its significantly easier to throw things up into the cloud.

All of the above, and more can account for some clutter. At Etsy, it meant that we knew of half the accounts we actually had, had maxed out the allotted number of s3 buckets and had individual administrative accounts for large chunks for the org. To give you a better idea about how things should REALLY be done - Most orgs only have a few number of admin and don’t let the rest of engineering touch AWS.

10 of 78

Evident.io

Scans of configurations to see if anything is misconfigured

Password policies?
Multi-factor Authentication
Jira Tickets

10

Evident.io

It was pretty difficult to figure out how to begin, but luckily Etsy had purchased a tool called Evident.io. Evident.io runs regular security scans to see if there is any misconfigured services. It details high/med/low risks in pretty big bold letters. It also gives you ways to remediate those those issues, whatever they might be.

So, first, I just started to look at the high risk issues. Most of them looked pretty serious because I was unfamiliar with a lot of the terminology, I had no idea what an ACL was at the time. Meh. I began by creating Jira tickets for all the high risk issues that I saw in Evident.io. Jira, is mostly just a way to organize what work needs to be done and by whom. I thought that would be a good way to begin organizing what was rattling around in my brain. Also, it would be a good way for other parts of the organization to take a look at what progress was being made and when I sent emails, I wanted something to point people to

11 of 78

11

Low-hanging IAM Fruit

# Password Policy

# Multi-factor auth

Monitoring

# Cloudtrail Logging

EC2

# Netwerkin’

# EC2 Roles

S3

# S3 Bucket Policies

12 of 78

Scout2

Github Page: https://github.com/nccgroup/Scout2
Reports for all accounts
Tie that into alerts manually

13 of 78

13

Low-hanging IAM Fruit

# Password Policy

# Multi-factor auth

Monitoring

# Cloudtrail Logging

EC2

# Netwerkin’

# EC2 Roles

S3

# S3 Bucket Policies

14 of 78

Changes I made… like a goon

Password policy to the highest scrutiny
Removed all admin roles from accounts that didn’t need them (aka hadn’t used aws in 2 yrs and didn’t have any api keys tied to their user)

14

15 of 78

Password Policies

16 of 78

My first Etsy communication

Hello X,

Looks like you still do not have MFA set up on your AWS account.

Go ahead and go to Identity & Access Management in your Amazon Web Services console -> find your username -> Manage MFA Device.

Note: If you no longer need your AWS account, please let me know!

Devina

17 of 78

Version 2.0

Hello X,

Looks like you still do not have MFA set up on your AWS account. It looks like you used your AWS account recently as well, so please sign up for MFA by 03/31/16 or your account will be suspended.

Go ahead and go to Identity & Access Management in your Amazon Web Services console -> find your username -> Manage MFA Device.

Note: If you no longer need your AWS account, please let me know!

Your neighborhood candy provider,

Devina

18 of 78

Oops...

19 of 78

Multiple statements which allow you to:

Resync MFA devices
Deactivate MFA devices
List MFA devices
Primary, management

Other policies:

Forcing MFA

So you will want to create a few different policies that allow your use to register for multi-factor auth.

This policy has many statements, and the reason is because everything in AWS Console has a bunch of dependencies. So you need to test your policies to verify that you have all the permissions to do what you wanna do. So here we have all the different things you can do with multi-factor auth. Resync, deactivate, list, and adding a device. As part of your statements, you will have optional ‘Sid’, just a discriptor of your statment. What effect you want it to have, allow/deny. Then what action you want your resources to be able to take. Here we have a conditional as well. You only want to be able to deactivate your mfa device, if you already have a MFA device.

Now, you may also want to put in place a multi-factor enforcement policy, thats pretty hardcore, it will prevent any user from proceeding to services if they have not enabled multi-factor auth.

20 of 78

Oof…

So, now that all that confusion was over and behind me. I was getting a better idea about how people were using AWS. I was able to take this information and create security groups. My original intention was to do role based access - aka - Given that you are a member of the dataeng team, you will likely use xyz services in aws. So, everyone in a particular team would have the same access. Unfortunately, hindesight is a thing - and I would have much prefered a methodology where it was more rulebased since such few people actually needed acess to aws. So if you needed full rights to EC2 or whatever, you would be in that group versus if you really only needed access to one instance. Something important to consider is that you shouldnt be using root api keys for all your stuff. So, create application specific keys so that you can rotate them without breaking other applications. Considering that no one likely looked at security before you, those root api keys will be prolific, so that stuff is gonna take forever to clean up.

21 of 78

Aws-cli for account creation

Becoming really used to the aws client is really useful too!

22 of 78

Using Terraform for IAM

What is terraform?
What can it do?

Static creds
Environment variables
Shared creds
EC2 Roles

23 of 78

Static Creds

24 of 78

24

Low-hanging IAM Fruit

# Password Policy

# Multi-factor auth

Monitoring

# Cloudtrail Logging

EC2

# Netwerkin’

# EC2 Roles

S3

# S3 Bucket Policies

25 of 78

Logging in AWS - Cloudtrail

26 of 78

27 of 78

28 of 78

29 of 78

ELK

30 of 78

Alert Types

Email:

Daily Roundup Emails

No production impacting

High Risk Alerts

Enough resources to handle

IRC/Slack/Jabber:

Slack & Dropbox

Collect the alerts:

Splunk
411 / Elastalert

31 of 78

31

Low-hanging IAM Fruit

# Password Policy

# Multi-factor auth

Monitoring

# Scout2

# Logging

EC2

# Netwerkin’

# EC2 Roles

S3

# S3 Bucket Policies

32 of 78

Inbound/Outbound

The next thing that was causing a lot of noise in Evident, was ports being open on EC2 instances. I saw that certain instances were entirely available on all ports, from all IP’s. Meaning that anyone around the world could interact with this instance on any port. In order to figure out what the next steps were, I talked to the network operations team about what our interanl ip space was. I also talked to the teams that were responsible for these instances who told me what ports would need to be open for these systems. Honestly though, figuring out who to talk to was the hardest part throughout the process. It was very difficult to figure out what team created an instance and what team owned an instance. So, I started tagging instances as i went for later use.

All in all, the general rule is, you want to make sure that only the people who SHOULD be able to access these systems, can.

33 of 78

EC2 Roles

Word had gotten around that there was someone on security who was diving head first into the land of aws. Engineers who were curious about the platform started asking me questions and advice with regards to the problems they were expeirencing in our current infrasture. They inquired about making instances, and when I was going through the process of creating an instance, I learned about EC2 roles. It was important for me to set up these EC2 roles on their behalf, so I was not limiting their usage of ec2 by having them talk to me every single time. Similar to what we discussed with regards to IAM policies, you can do the same for EC2 instances. So what permissions does this one particular instance need? It is important that you set your instances up with roles to begin with because you can’t set them up as an after thought.

34 of 78

34

Low-hanging IAM Fruit

# Password Policy

# Multi-factor auth

Monitoring

# Scout2

# Logging

EC2

# Netwerkin’

# EC2 Roles

S3

# S3 Bucket Policies

35 of 78

Bucket Policies

36 of 78

37 of 78

38 of 78

Bug Bounties at Etsy: https://www.etsy.com/bounty
S3 Scanner �Github: https://github.com/bear/s3scan

Report of all s3 buckets and perms
Likely how bountiers are finding out about your misconfigured policies.

39 of 78

39

So… it happened, what do I do now?

Write down all the systems you need to take care of
Find out what you need to fix on all systems, write that down
Start with the low-hanging fruit
Over communicate what you are doing.
Work with networking on the AWS network
Create default rulesets & roles
Work with IT/helpdesk to handle account provisioning
Work with systems engineering to handle provisioning of services
… profit?

40 of 78

THANKS!

3@etsy.com

@theulzo

41 of 78

Introduction

41

Etsy (Jan 2015 - Present)
Orbitz (May 2014 - Dec 2015)
University of Illinois in Chicago

42 of 78

Etsy operates a global marketplace where people around the world connect,

both online and offline, to make, sell and buy unique goods.

42

43 of 78

Security at Etsy

Evangelizing Security at Etsy

Candy is a great way to make friends
Allow the conversation about security to be comfortable and inviting.

43

Security at Etsy is radical in that at its inception was already widely evangelized throughout the organization. Everywhere from product to systems engineering, security is major part of decisions that are made and it is even difficult to find a human that doesn't view security in a positive light.

Mostly because of a program we call security-candy, where we make candy available for the people who work at Etsy. This program exists to create a comfortable association with security and the people on the team. So, humans will come around to grab candy and may sometimes even have questions for us or will come by to talk to us about somehting they are working on. The relationships this candy builds is one low-cost method of evangelizing security.

Though, that is a gross over simplification for work that was done by the creators of the security team. So, as you can tell, Etsy is a magical place filled with unicorn engineers and is also a beautiful place to work.

AND WE ARE HIRING! We are hiring for managers, senior engineers and engineers for the Brooklyn office and for Remote work. I am a remote employee for Etsy and I love it.

44 of 78

What is this talk about?

I will help you improve your existing AWS infrastructure
You will walk away with action items
http://bit.ly/2EnZU1q

44

45 of 78

“Securing Amazon Web Services”

45

46 of 78

Traditional bare metal
Minimal footprint in the clouds

Infrastructure

46

47 of 78

Where to begin?

Previously I had mentioned how large the scope of this project was when it was handed to me, the reason is because at any growing company, you will have humans that will want to test things in various infrastructures. You go through acquisitions. You may need to run external scans of your infrastructure as well. And especially if you want contractors to build sites for you, you can’t just give them access to your entire infra. So in so many ways, its significantly easier to throw things up into the cloud.

All of the above, and more can account for some clutter. At Etsy, it meant that we knew of half the accounts we actually had, had maxed out the allotted number of s3 buckets and had individual administrative accounts for large chunks for the org. To give you a better idea about how things should REALLY be done - Most orgs only have a few number of admin and don’t let the rest of engineering touch AWS.

48 of 78

Evident.io

Scans of configurations to see if anything is misconfigured

Password policies?
Multi-factor Authentication
Jira Tickets

48

Evident.io

It was pretty difficult to figure out how to begin, but luckily Etsy had purchased a tool called Evident.io. Evident.io runs regular security scans to see if there is any misconfigured services. It details high/med/low risks in pretty big bold letters. It also gives you ways to remediate those those issues, whatever they might be.

So, first, I just started to look at the high risk issues. Most of them looked pretty serious because I was unfamiliar with a lot of the terminology, I had no idea what an ACL was at the time. Meh. I began by creating Jira tickets for all the high risk issues that I saw in Evident.io. Jira, is mostly just a way to organize what work needs to be done and by whom. I thought that would be a good way to begin organizing what was rattling around in my brain. Also, it would be a good way for other parts of the organization to take a look at what progress was being made and when I sent emails, I wanted something to point people to

49 of 78

49

Low-hanging IAM Fruit

# Password Policy

# Multi-factor auth

Monitoring

# Cloudtrail Logging

EC2

# Netwerkin’

# EC2 Roles

S3

# S3 Bucket Policies

50 of 78

Scout2

Github Page: https://github.com/nccgroup/Scout2
Reports for all accounts
Tie that into alerts manually

51 of 78

51

Low-hanging IAM Fruit

# Password Policy

# Multi-factor auth

Monitoring

# Cloudtrail Logging

EC2

# Netwerkin’

# EC2 Roles

S3

# S3 Bucket Policies

52 of 78

Changes I made… like a goon

Password policy to the highest scrutiny
Removed all admin roles from accounts that didn’t need them (aka hadn’t used aws in 2 yrs and didn’t have any api keys tied to their user)

52

53 of 78

Password Policies

54 of 78

My first Etsy communication

Hello X,

Looks like you still do not have MFA set up on your AWS account.

Go ahead and go to Identity & Access Management in your Amazon Web Services console -> find your username -> Manage MFA Device.

Note: If you no longer need your AWS account, please let me know!

Devina

55 of 78

Version 2.0

Hello X,

Looks like you still do not have MFA set up on your AWS account. It looks like you used your AWS account recently as well, so please sign up for MFA by 03/31/16 or your account will be suspended.

Go ahead and go to Identity & Access Management in your Amazon Web Services console -> find your username -> Manage MFA Device.

Note: If you no longer need your AWS account, please let me know!

Your neighborhood candy provider,

Devina

56 of 78

Oops...

57 of 78

Multiple statements which allow you to:

Resync MFA devices
Deactivate MFA devices
List MFA devices
Primary, management

Other policies:

Forcing MFA

So you will want to create a few different policies that allow your use to register for multi-factor auth.

This policy has many statements, and the reason is because everything in AWS Console has a bunch of dependencies. So you need to test your policies to verify that you have all the permissions to do what you wanna do. So here we have all the different things you can do with multi-factor auth. Resync, deactivate, list, and adding a device. As part of your statements, you will have optional ‘Sid’, just a discriptor of your statment. What effect you want it to have, allow/deny. Then what action you want your resources to be able to take. Here we have a conditional as well. You only want to be able to deactivate your mfa device, if you already have a MFA device.

Now, you may also want to put in place a multi-factor enforcement policy, thats pretty hardcore, it will prevent any user from proceeding to services if they have not enabled multi-factor auth.

58 of 78

Oof…

So, now that all that confusion was over and behind me. I was getting a better idea about how people were using AWS. I was able to take this information and create security groups. My original intention was to do role based access - aka - Given that you are a member of the dataeng team, you will likely use xyz services in aws. So, everyone in a particular team would have the same access. Unfortunately, hindesight is a thing - and I would have much prefered a methodology where it was more rulebased since such few people actually needed acess to aws. So if you needed full rights to EC2 or whatever, you would be in that group versus if you really only needed access to one instance. Something important to consider is that you shouldnt be using root api keys for all your stuff. So, create application specific keys so that you can rotate them without breaking other applications. Considering that no one likely looked at security before you, those root api keys will be prolific, so that stuff is gonna take forever to clean up.

59 of 78

Aws-cli for account creation

Becoming really used to the aws client is really useful too!

60 of 78

Using Terraform for IAM

What is terraform?
What can it do?

Static creds
Environment variables
Shared creds
EC2 Roles

61 of 78

Static Creds

62 of 78

62

Low-hanging IAM Fruit

# Password Policy

# Multi-factor auth

Monitoring

# Cloudtrail Logging

EC2

# Netwerkin’

# EC2 Roles

S3

# S3 Bucket Policies

63 of 78

Logging in AWS - Cloudtrail

64 of 78

65 of 78

66 of 78

67 of 78

ELK

68 of 78

Alert Types

Email:

Daily Roundup Emails

No production impacting

High Risk Alerts

Enough resources to handle

IRC/Slack/Jabber:

Slack & Dropbox

Collect the alerts:

Splunk
411 / Elastalert

69 of 78

69

Low-hanging IAM Fruit

# Password Policy

# Multi-factor auth

Monitoring

# Scout2

# Logging

EC2

# Netwerkin’

# EC2 Roles

S3

# S3 Bucket Policies

70 of 78

Inbound/Outbound

The next thing that was causing a lot of noise in Evident, was ports being open on EC2 instances. I saw that certain instances were entirely available on all ports, from all IP’s. Meaning that anyone around the world could interact with this instance on any port. In order to figure out what the next steps were, I talked to the network operations team about what our interanl ip space was. I also talked to the teams that were responsible for these instances who told me what ports would need to be open for these systems. Honestly though, figuring out who to talk to was the hardest part throughout the process. It was very difficult to figure out what team created an instance and what team owned an instance. So, I started tagging instances as i went for later use.

All in all, the general rule is, you want to make sure that only the people who SHOULD be able to access these systems, can.

71 of 78

EC2 Roles

Word had gotten around that there was someone on security who was diving head first into the land of aws. Engineers who were curious about the platform started asking me questions and advice with regards to the problems they were expeirencing in our current infrasture. They inquired about making instances, and when I was going through the process of creating an instance, I learned about EC2 roles. It was important for me to set up these EC2 roles on their behalf, so I was not limiting their usage of ec2 by having them talk to me every single time. Similar to what we discussed with regards to IAM policies, you can do the same for EC2 instances. So what permissions does this one particular instance need? It is important that you set your instances up with roles to begin with because you can’t set them up as an after thought.

72 of 78

72

Low-hanging IAM Fruit

# Password Policy

# Multi-factor auth

Monitoring

# Scout2

# Logging

EC2

# Netwerkin’

# EC2 Roles

S3

# S3 Bucket Policies

73 of 78

Bucket Policies

74 of 78

75 of 78

76 of 78

Bug Bounties at Etsy: https://www.etsy.com/bounty
S3 Scanner �Github: https://github.com/bear/s3scan

Report of all s3 buckets and perms
Likely how bountiers are finding out about your misconfigured policies.

77 of 78

77

So… it happened, what do I do now?

Write down all the systems you need to take care of
Find out what you need to fix on all systems, write that down
Start with the low-hanging fruit
Over communicate what you are doing.
Work with networking on the AWS network
Create default rulesets & roles
Work with IT/helpdesk to handle account provisioning
Work with systems engineering to handle provisioning of services
… profit?

78 of 78

THANKS!

3@etsy.com

@theulzo