Final review

CS224p

1

Key Languages and Frameworks we Learned

• MapReduce
• Spark

2

MapReduce

3

https://www.edureka.co/blog/mapreduce-tutorial/

Word count

4

5

6

Hadoop Main Components

HDFS (Hadoop Distributed File System): This is the storage component of Hadoop, which allows for the storage of large amounts of data across multiple machines. It is designed to work with commodity hardware, which makes it cost-effective.

YARN (Yet Another Resource Negotiator): This is the resource management component of Hadoop, which manages the allocation of resources (such as CPU and memory) for processing the data stored in HDFS.

Hadoop is commonly used in big data scenarios such as data warehousing, business intelligence, and machine learning. It’s also used for data processing, data analysis, and data mining. It enables the distributed processing of large data sets across clusters of computers using a simple programming model.

​

7

8

9

10

MapReduce Code

11

public static class Map extends Mapper<LongWritable,Text,Text,IntWritable> {

     public void map(LongWritable key, Text value, Context context) throws IOException,InterruptedException {

          String line = value.toString();

          StringTokenizer tokenizer = new StringTokenizer(line);

          while (tokenizer.hasMoreTokens()) {

                value.set(tokenizer.nextToken());

                context.write(value, new IntWritable(1));

           }

public static class Reduce extends Reducer<Text,IntWritable,Text,IntWritable> {

     public void reduce(Text key, Iterable<IntWritable> values,Context context)

           throws IOException,InterruptedException {

                int sum=0;

                for(IntWritable x: values)

                {

                      sum+=x.get();

                }

                context.write(key, new IntWritable(sum));

           }

      }

MapReduce

• Advantages and Disadvantages?
• +: simplicity
• -: high I/O cost

12

Resilient Distributed Datasets

RDD is a fundamental data structure in Apache Spark. It is an immutable distributed collection of objects that can be processed in parallel across a cluster of machines. RDDs provide fault tolerance, meaning they can recover from failures, and they enable efficient distributed data processing.

​

• Immutable: RDDs are read-only, meaning their data cannot be modified once created. However, you can transform an existing RDD into a new RDD through various operations.
• Distributed: RDDs are distributed across multiple nodes in a cluster, allowing for parallel processing. The data in RDDs is partitioned and stored in memory or on disk across the worker nodes.
• Resilient: RDDs are fault-tolerant. If a node fails, RDDs can recover by recomputing the lost partitions using lineage information—the series of transformations that were applied to the RDD. This ensures data reliability and fault tolerance.
• Lazy Evaluation: RDDs support lazy evaluation, which means that transformations on RDDs are not executed immediately. Instead, Spark keeps track of the applied transformations as a directed acyclic graph (DAG) of operations. The actual computation is performed when an action operation is called, triggering the execution of the entire DAG.
• In-Memory Processing: RDDs provide an option to persist data in memory, enabling fast iterative and interactive data processing. This feature allows Spark to achieve high-performance processing by minimizing disk I/O.

​

13

14

Spark

RDD Actions versus Transformations

RDDs support two types of operations:

• transformations, which create a new dataset from an existing one, and
• actions, which return a value to the driver program after running a computation on the dataset.
• For example, map() is a transformation, reduce() is an action
• All transformations in Spark are lazy
• What is advantage of lazy?

15

Spark -intro

• RDDread = sc.textFile (file:///opt/spark/CHANGES.txt)
• RDDread. Collect ()

​

16

• RDDread. First ()

RDDread. Take (5)

https://www.dezyre.com/apache-spark-tutorial/pyspark-tutorial

Spark DataFrame

• dataframe_mysql = sqlContext.read.format("jdbc").options( url="jdbc:mysql://:3306/demo",driver = "com.mysql.jdbc.Driver",dbtable = "demotable",user="root", password="XXXXX").load() dataframe_mysql.show()

+---+-----+-------+

| id| name|age|

+---+-------+---+

|  1|abhay| 25|

|  2|banti| 25|

+---+------+---+

17

Spark - map

18

confusedRDD = sc.textFile("confused.txt")

mappedconfusion = confusedRDD.map(lambda line : line.split(" "))

mappedconfusion.take(5)

Spark - flatMap

19

flatMappedConfusion = confusedRDD.flatMap(lambda line : line.split(" "))

flatMappedConfusion.take(5)

onlyconfusion = confusedRDD.filter(lambda line : ("confus" in line.lower()))

onlyconfusion.count()

3

Spark - reduceByKey

20

data = [("a", 3), ("b", 4), ("a", 1)]

sc.parallelize(data).reduceByKey(lambda x, y: x + y) # Default parallelism

sc.parallelize(data).reduceByKey(lambda x, y: x + y, 10) # Custom parallelism

Given a collection of names and charges, compute total charge per name

21

22

Links

https://www.youtube.com/watch?v=aReuLtY0YMI

​

23