1 of 23

Google File System / Hadoop

Alan Nguyen, Tom Lai

University of Washington

View this presentation at bit.ly/hadoop434

2 of 23

Outline

  • Intro
  • Architecture
    • Namenode
    • Datanode
  • Availability/Recovery
  • Consistency/Atomicity
  • Application domains focused on by this system
  • Other notable features such as performance, scalability, and security
  • Paper Critique

View this presentation at bit.ly/hadoop434

3 of 23

Intro

Hadoop Features

  • Open-source Java Framework
    • supported on all major platforms
  • Extremely fault-tolerant
    • hardware failure is the norm rather than the exception
  • Slaves run on cheap computers
  • Linear Scalability

​

View this presentation at bit.ly/hadoop434

4 of 23

Intro

the idea came from google

  • A paper was published
  • But Google's implementation (Google file system) was private to google
  • Doug Cutting read the paper and used the architecture in his project
    • calls it Hadoop
  • GFS and HDFS is very similar

View this presentation at bit.ly/hadoop434

5 of 23

Intro

Fun Fact

“The name my kid gave a stuffed yellow elephant. Short, relatively easy to spell and pronounce, meaningless, and not used elsewhere: those are my naming criteria. Kids are good at generating such. Googol is a kid’s term”

  • Doug Cutting

View this presentation at bit.ly/hadoop434

6 of 23

Intro

Who use Hadoop

(and more)

View this presentation at bit.ly/hadoop434

7 of 23

Architecture

Components

1. NameNode (master)

​

2. DataNode (slave)

​

3. Client

​

View this presentation at bit.ly/hadoop434

8 of 23

Architecture

DataNode

  • Slave (many)�
  • Stores data�
  • Compute data
    • when node is also acting as TaskTracker

​

​

​

���

​

​

​

View this presentation at bit.ly/hadoop434

9 of 23

Architecture

NameNode

  • Master (only one per cluster)�
  • Stores metadata
    • Which DataNodes are alive
    • Which DataNodes have which data blocks�
  • Interact with client
    • Accepts requests
    • Redirect to DataNodes�

​

​

​

View this presentation at bit.ly/hadoop434

10 of 23

View this presentation at bit.ly/hadoop434

11 of 23

View this presentation at bit.ly/hadoop434

12 of 23

View this presentation at bit.ly/hadoop434

13 of 23

View this presentation at bit.ly/hadoop434

14 of 23

View this presentation at bit.ly/hadoop434

15 of 23

Availability / Recovery

Checkpoint Node / Secondary Namenode

  • Merges edit logs into the image file
  • Sends it to namenode
  • Problem: May take a long time for Checkpoint node to start
    • has to get heartbeat message from DataNode, and block locations
    • hours for large clusters
    • Hadoop 2.x uses Standby Node

View this presentation at bit.ly/hadoop434

16 of 23

Availability / Recovery

Checkpointing

  • Snapshot of file system is stored in:
    • fsimage
    • edit log
  • Changing fsimage directly is network and I/O intensive, so better to write new edit log, and periodically merge them

​

View this presentation at bit.ly/hadoop434

17 of 23

Availability / Recovery

Clones of Blocks

  • Each file is divided into blocks

​

  • Multiple copies of each blocks stored on multiple nodes

​

View this presentation at bit.ly/hadoop434

18 of 23

Availability / Recovery

Rack Distribution

  • Files are divided into blocks

​

  • Copies of each block are placed on separate racks

​

  • If one rack fails, the other racks will have copies of all blocks of that file

View this presentation at bit.ly/hadoop434

19 of 23

Availability

​

Heartbeat Messages

  1. The master receives a heartbeat message from slaves periodically
  2. No heartbeat = dead 💀💀💀
    1. no new IO requests for dead nodes
    2. make new replicas if not enough
  3. Each heartbeat message also includes what file blocks they have
  4. The master acknowledge each heartbeat messages
    • sometimes a command is piggybacked on the ack

View this presentation at bit.ly/hadoop434

20 of 23

Consistency / Atomicity enforcement

These must be atomic:

  • Creating a file
  • Deleting file
  • Renaming file
  • Renaming directory
  • Creating directory

​

Deleting directory recursively may not be atomic

One-copy-update semantics

  • every read sees the effect of all previous writes

All immediately visible:

  • CRUD -- create, rename, update, delete
  • Delete followed by create�

​

Consistency

Atomicity

View this presentation at bit.ly/hadoop434

21 of 23

Application domains focused on by this system

Data Collection

  • Log storage
  • Data storage
    • Documents
    • Sensors

Data Processing

  • Map Reduce
  • Analyze user behavior

​

Machine learning

  • Spam Filters
  • Generate suggestions

​

View this presentation at bit.ly/hadoop434

22 of 23

Other notable features

DataNode Hot Swap Drive

  • You can add or replace harddrives without shutting down the datanode

Checkpoint Node

  • Make backups of the Namenode

View this presentation at bit.ly/hadoop434

23 of 23

Paper Critique

​

GFS

GFS Case Study

Hadoop

Contributions

GFS Architecture and write examples/ lease examples

Mentions the limitation of having one GFS instance per data center and how they worked around it

Hadoop Architecture-- very similar to GFS.

A lot of information, more than you can read.

Drawbacks of ideas

  • Single master was not scalable enough (limit in memory size). An issue addressed in the GFS Case study.
  • Some problems doesn't fit neatly into a single Map/Reduce job

How to improve

  • Hadoop uses a different file system, you have to move it to hadoop's file space to do any computation, and then move it back to see the result

View this presentation at bit.ly/hadoop434