10-605 AWS Setup
Today’s Goal: Create a DIY supercomputer
Quota increase request
Quota increase request
Search EC2 and select this
Quota increase request
Normally, we use G and VT instances (depend on your instance type).
You can choose either a spot instance or an on-demand instance based on your needs. �Spot instances are cheaper but may be terminated if there is a on-demand request. On-demand instances are more expensive but stable, as they cannot be occupied by others.
Select the type that suits your needs. We recommend requesting both instances, as you may use both of them.
You may have a value of 0 for both at this time.
Quota increase request
Take on-demand request as an example
Quota increase request
Take on-demand request as an example
Enter the amount you want. We recommend 8.
By clicking “Request” you are submitting your request.
It may take anywhere from a few hours to a few days to process, so be sure to request early!
Quota increase request
Take on-demand request as an example
Once you receive your quotas, you will be able to see your quotas here!
10-605 Setup for Homework
Credits and Budget Tips
Checking Credits
To check your credits, from your AWS Main Page:
IMPORTANT: For 10-605/805 make sure you are connected to N. Virginia US-EAST-1
Checking Credits
Normally you should have $150. I have $300 because I have two courses.
Checking Amount Spent
Checking Amount Spent
IMPORTANT: Creating a Budget Warning
Creating a Budget Warning
Creating a Budget Warning
Give your budget a name
Since the assignments are only over 1 month select “Monthly”
You can choose “Expiring” and select the appropriate start and end month
Choose “Fixed” and since the assignment can be completed with $30 I recommend using this as your Budgeted amount.
Creating a Budget Warning
Creating a Budget Warning
Set up an appropriate threshold of your budgeted amount. I used 80%
Add your email address
You can change this to “Forecasted” costs but this might cause AWS to warn you earlier than necessary, I’ll keep it on “Actual”
Creating a Budget Warning
Creating a Budget Warning
Creating a Budget Warning
10-605 AWS Setup
10-605 Setup for Homework
Setup S3 Bucket
Setup S3 Bucket
Setup S3 Bucket
Give your bucket a name. This name must be globally unique and not contain spaces or uppercase letters
Then click: “Create Bucket”
Setup S3 Bucket
Setup S3 Bucket - Adding Custom Bootstrap Action Script to Your Bucket
Later when we create an EMR Cluster we will need to set up some extra packages to be pip installed. Using bootstrapping allows us to start the cluster with these preinstalled!
#!/bin/bash
sudo python3 -m pip install matplotlib pillow boto3 pandas
Setup S3 Bucket - Adding Custom Bootstrap Action Script to Your Bucket
Click on the S3 bucket you created:
Setup S3 Bucket - Adding Custom Bootstrap Action Script to Your Bucket
Click the upload button:
Setup S3 Bucket - Adding Custom Bootstrap Action Script to Your Bucket
Drop your .sh file you created onto the page, then click the ‘Upload’ button in the bottom right.
Setup S3 Bucket - Adding Custom Bootstrap Action Script to Your Bucket
Your file should appear there:
Click on the the file.
Setup S3 Bucket - Adding Custom Bootstrap Action Script to Your Bucket
Click “Copy S3 URI” and paste it somewhere, you will need this later!
10-605 Setup for Homework
Setup IAM Roles
Setup IAM Roles
You might not have any current roles
Setup IAM Roles
Setup IAM Roles
Search for “S3FullAccess”
Select “AmazonS3FullAccess”
This is allowing EC2 instances (which you will use in future homework) to have full access to all S3 Buckets.
Setup IAM Roles
Give your role a name
Setup IAM Roles
EMR Role Setup
Click Roles
EMR Role Setup
Select AWS Service
Select EMR
From the drop down menu
Search EMR
EMR Role Setup
Select AWS Service
Select EMR
From the drop down menu
Search EMR
EMR Role Setup
Select Next
EMR Role Setup
Name your role, then click Create role at the bottom of the page
EMR Role Setup
Select your newly created EMR role
Setup IAM Roles
Select add permissions and attach policies
Setup IAM Roles
Search “AmazonElasticMapReduceRole” and select it
Add permission
Setup IAM Roles
10-605 Setup for Homework
Make a keypair
Make a keypair
Click “Key Pairs”
Then click “Create key pair”
Make a keypair
Give your key a name, I recommend not using spaces
Select RSA
Select .pem
Make a keypair
Make a keypair - DONE Kind of…
Make a keypair - Windows Instructions
Make a keypair - Windows Instructions
Make a keypair - Windows Instructions
Make a keypair - Windows Instructions
Make a keypair - Mac Instructions
10-605 Setup for Homework
Setup Security Groups
Setup Security Groups
Select your “default” security group
Click “Inbound rules” in the window here
Click “Edit inbound rules”
Setup Security Groups
Click Add Rule
Delete any existing rules
Then add both of these rules.
MY IP
MY IP
Click “Save rules”.
10-605 Setup for Homework
Setup EMR Cluster
Setup EMR Cluster
Setup EMR Cluster
Setup EMR Cluster
Name your cluster, for release we are using emr-6.9.0
Select Custom!
Setup EMR Cluster
Hadoop, JupyterHub and Spark, no other choices should be selected
Setup EMR Cluster
While you are still developing your notebook I recommend using just 1 Core m4.xlarge instance, you can increase number and power of the machine once you have worked it all through on the subsample of the data and you want to run it on the entire dataset.
Maybe 2-4, m4.xlarge, be sure to keep an eye on your spending
Setup EMR Cluster
Remove task instance group as we will not need it
Setup EMR Cluster
You can increase the number of core (worker) instances here, as mentioned, we recommend using 1 when writing your homework to save on costs and bumping it up when you are ready to run it on the full dataset.
Setup EMR Cluster
Under Networking and EC2 Security Groups (Firewall)
For both the Primary and Core Nodes dropdowns select your default security group which you created earlier
Setup EMR Cluster
It’s up to you if you would like cluster termination. We recommend having it set to Manually terminate cluster, to avoid you accidentally leaving your cluster idle for an hour and having your homework progress being deleted. Just be sure to terminate it when you are not working on your homework so you don’t get charged
Setup EMR Cluster
We need to add a bootstrap Action. By adding the .sh file which we stored in our S3 bucket earlier
Setup EMR Cluster
Name your Bootstrap action whatever you would like.
Then add the Script location we got from the S3 bucket when we added the script using the ‘Copy S3 URI’ button we used.
Setup EMR Cluster
Uncheck “Publish cluster-specific logs to Amazon S3”
Setup EMR Cluster
Add the following to the Software settings. (Copy from the next slide)
Edit Software Setting
[
{
"Classification": "spark",
"Properties": {
"maximizeResourceAllocation": "true",
"spark.executor.memory": "16G",
"spark.driver.memory": "16G"
}
},
{
"Classification": "jupyter-s3-conf",
"Properties": {
"s3.persistence.enabled": "true",
"s3.persistence.bucket": "YOUR-OWN-BUCKET_NAME"
}
}
]
Change these two to reflect your Primary and Core memory values. We chose m4.xlarge for both which both had 16G
Change it to your own bucket name
Setup EMR Cluster
Browse the for the EC2 Key pair which you created.
For these two, head to slide 27 for instructions on how to create the service role and the Instance profile role! Attach them here.
Once you are done here press Create Cluster
Setup EMR Cluster
Now the instance will start, keep an eye on the status here, it should take a while bootstrapping.
Setup EMR Cluster
Now that it is ‘Waiting’ we are good to go!
10-605 Setup for Homework
Get Access to Jupyter Notebooks Interface
Click Applications
Click this link
Get Access to Jupyter Notebooks Interface
Click Continue to ec2-....
Get Access to Jupyter Notebooks Interface
Click Continue to ec2-....
Get Access to Jupyter Notebooks Interface
Get Access to Jupyter Notebooks Interface
When you are in your notebook make sure that you change the Kernel to “PySpark”
Get Access to Jupyter Notebooks Interface
All set!
10-605 Setup for Homework
Terminate and Clone EMR Cluster
Terminate the cluster
Terminate and Clone EMR Cluster
Click Terminate
After a few minutes, you can see the status is terminated. (Please make sure you successfully terminate the cluster!)
Terminate and Clone EMR Cluster
Select the cluster you’d like to clone and click “clone”
Terminate and Clone EMR Cluster
You can update the configurations and then click “clone cluster” to quickly start a cluster.
10-605 Setup for Homework
How about GCP? Aren’t we using that for HW3?
Check out Sam Zhou’s great presentation here:
drive.google.com/file/d/1igMJDn5yqXdT_FT8sQYe-nUvLAfgA581/view?usp=sharing
Huge thank you to the Fall 2025 TA Team!!
Special thanks to Baihong for updating this slide deck