2 of 26

Contents

Integrity
Compression
Serialization
File-based Data Structure

�

3 of 26

Data Integrity

I/O operation on the disk or network carries with it a small chance of introducing errors into the data
When the volumes of data flowing through the system are as large as the ones Hadoop is capable of handling, the chance of data corruption occurring is high

Checksum

Usual way of detecting corrupted data
Technique for only error detection (cannot fix the corrupted data)
CRC-32 (cyclic redundancy check)
Compute a 32-bit integer checksum for input of any size

Usual way of detecting corrupted data

–

Technique for only error detection (cannot fix the corrupted data)

–

CRC

32 (cyclic redundancy check)



Compute a 32

bit integer checksum for input of any size

/ 18

�

4 of 26

Data Integrity in HDFS

HDFS transparently checksums all data written to it and by default verifies checksums when reading data.
io.bytes.per.checksum

Data size to compute checksums is 512 bytes
CRC-32C checksum is 4 bytes long

Datanodes are responsible for verifying the data they receive before storing the data and its checksum

If it detects an error, the client receives a ChecksumException, a subclass of IOException

When clients read data from datanodes, they verify checksums as well, comparing them with the ones stored at the datanode

5 of 26

Data Integrity in HDFS

Checksum verification log

Each datanode keeps a persistent log to know the last time each of its blocks was verified. When a client successfully verifies a block, it tells the datanode who sends the block. Then, the datanode updates its log

6 of 26

DataBlockScanner

Each datanode runs a DataBlockScanner in a background thread that periodically verifies all the blocks stored on the datanode.
Guard against corruption due to “bit rot” in the physical storage media
Healing corrupted blocks

If a client detects an error when reading a block, it reports the bad block and the datanode to the namenode
Namenode marks the block replica as corruptNamenode schedules a copy of the block to be replicated on another datanode
The corrupt replica is deleted
Disabling verification of checksumPass false to the setVerifyCheckSum() method on FileSystem-ignoreCrc option

7 of 26

Disabling verification of checksum

Pass false to the setVerifyCheckSum() method on FileSystem before using the open() method to read a file.
-IgnoreCrc option

File’s checksum can be with hadoop fs -checksum. This is useful to check whether two files in HDFS have the same contents

8 of 26

LocalFileSystem

Performes client-side checksumming
When you write a file called filename, the FS client transparently creates a hidden file, .filename.crc, in the same directory containing the checksums for each chunk of the file
You can directly create a RawLocalFileSystem instance, which may be useful if you want to disable checksum
verification

9 of 26

ChecksumFileSystem

Wrapper around FileSystem
LocalFileSystem uses ChecksumFileSystem to do its work
Make it easy to add checksumming to other (nonchecksummed) FS
Underlying FS is called the raw FSFileSystem

rawFs = ...FileSystem checksummedFs = new ChecksumFileSystem(rawFs)

If an error is detected by ChecksumFileSystem when reading a file, it will call its reportChecksumFailure() method.

10 of 26

Compression

Two major benefits of file compression

Reduce the space needed to store files
Speed up data transfer across the network
When dealing with large volumes of data, both of these savings can be significant, so it pays to carefully consider how to use compression in Hadoop

Compression formats

11 of 26

Splittable Colomns

Indicates whether the compression format supports splitting
Whether you can seek to any point in the stream and start reading from
Splittable compression formats are especially suitable for MapReduce

12 of 26

Implementation of a compression decompression algorithm

The LZO libraries are GPLlicensed and may not be included in Apache distributions

13 of 26

CompressionCodec�

CreateOutputStream(OutputStreamout):

create a CompressionOutputStream to which you write your uncompressed data to have it written in compressed form to the underlying stream.

CreateInputStream(InputStreamin):

obtain a CompressionInputStream, which allows you to read uncompressed data from the underlying stream.

15 of 26

Serialization

Process of turning structured objects into a byte stream for transmission over a network or for writing to persistent storage.
Deserialization is the reverse process of serialization.
Used in two quite distinct areas of distributed data processing:

For interprocess communication
For persistent storage

Interprocess communication between nodes in the system is implemented using remote procedure calls (RPCs).

16 of 26

Requirements of RPC

Compact

To make efficient use of storage space

Fast

The overhead in reading and writing of data is minimal

Extensible

We can transparently read data written in an older format

Interoperable

We can read or write persistent data using different language

Hadoop uses its own serialization format, Writables,

17 of 26

The Writable Interface

Writable interface defines two methods

write() for writing its state to a DataOutput binary stream
readFields() for reading its state from a DataInput binary stream

public interface Writable {

void write(DataOutput out) throws IOException;

void readFields(DataInput in) throws IOException;

}

IntWritable writable = new IntWritable();writable.set(163);

public static byte[] serialize(Writable writable) throws IOException {

ByteArrayOutputStream out = new ByteArrayOutputStream();

DataOutputStream dataOut = new DataOutputStream(out);

writable.write(dataOut);

dataOut.close();

return out.toByteArray();

}

byte[] bytes = serialize(writable);

18 of 26

Writable Classes

19 of 26

Text Writable

Can be thought of as the Writable equivalent of java.lang.String
Replacement for the org.apache.hadoop.io.UTF8 class (deprecated)
Maximum size is 2GB
Use standard UTF-8
org.apache.hadoop.io.UTF8 used Java’s modified UTF-8
Indexing for the Text class is in terms of position in the encoded byte sequence
Text is mutable (like all Writable implementations, except NullWritable)
You can reuse a Text instance by calling one of the set() method

Text t = new Text("hadoop");

t.set("pig");

assertThat(t.getLength(), is(3));

assertThat(t.getBytes().length, is(3));

20 of 26

BytesWritable

Wrapper for an array of binary data

NullWritable

Zero-length serialization
Used as a placeholder
A key or a value can be declared as a NullWritable when you don’t need to use that position

ObjectWritable

General-purpose wrapper for Java primitives, String, enum, Writable, null, arrays of any of these types
Useful when a field can be of more than one type

Writable collections

ArrayWritable
TwoDArrayWritable
MapWritable
SortedMapWritable

21 of 26

WritableComparable and comparators

Comparison of types is crucial for MapReduce, where there is a sorting phase during which keys are compared with one another.
One optimization that Hadoop provides is the RawComparator extension of Java’s Comparator:

package org.apache.hadoop.io;

import java.util.Comparator;

public interface RawComparator<T> extends Comparator<T> {

public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2);

}

For example, the comparator for IntWritables implements the raw compare() method by reading an integer from each of the byte arrays b1 and b2 and comparing them directly

22 of 26

Serialization Frameworks

Hadoop has an API for pluggable serialization frameworks.
A serialization framework is represented by an implementation of Serialization. WritableSerialization, for example, is the implementation of Serialization for Writable types.
A Serialization defines a mapping from types to Serializer instances (for turning an object into a byte stream) and Deserializer instances (for turning a byte stream into an object).
Serialization frameworks approaches a problem by defining them in a language neutral, declarative fashion, using an interface description language (IDL).
Apache Thrift , Google Protocol Buffers, Avro are both popular serialization frameworks and both are commonly used as a format for persistent binary data.

23 of 26

File-Based Data Structures

A file format is just a way to define how information is stored in HDFS file system.
Hadoop-focused file formats to use for structured and unstructured data.
Sequence files and Map files are the binary file formats.
Avro datafiles are like sequence files in that they are designed for large-scale data processing—they are compact and splittable—but they are portable across different programming languages.
Sequence files, map files, and Avro datafiles are all row-oriented file formats

24 of 26

Sequence file

Provides a persistent data structure for binary key-value pairs.
Sequence files were originally designed for MapReduce, so the integration is smooth
They encode a key and a value for each record. Records are stored in a binary format that is smaller than a text-based format.
They support block-level compression, so you can compress the contents of the file while also maintaining the ability to split the file into segments for multiple map tasks.

1 of 26