MapReduce

Map - Reduce:

   Programming model for data processing.

Data Processing :

 operations on data ,m especially by a computer, to retrieve, transform or classify information.

Batch Processing: 

execution of series of programs(jobs) on a computer without any manual intervention. jobs are set up so that they can be run to completion without human interaction.All input parameters are predefined through scripts, command line arguments, control files or job control language.

Hadoop uses parallel processing of data.

Map- Reduce: there are two phases in the Map - Reduce:

The Map Phase and the Reduce Phase. Each phase accepts input and output in key value pairs. so there are two functions

 1. Map Function

 2. Reduce Function.

Map Function extracts the relevant data and then places in the ascending order. Here the keys are ignored, only data is extracted and placed in ascending order. this is the output of the map function.

The output of the map function is used as input of the reduce function. the reduce function reduces the input and the final output is given. the below is the example:

Input to the map function as text format:
(harry,82484Boston)

(harry,298246America)

(albus,23429London)

(albus,23145Britain)

Output of the Map function is :

(harry,Boston)

(harry,America) 

(albus,London)

(albus,Britain)

the output of the map function is a feed to the Reduce function as a input. then the reduce function will produce the output in the r educed format of the data:

(harry,[Boston,America])

(albus,[London,Britain])

Mapper class has four parameters : input key , input value and output key, output value.

Hadoop basic types : 

LongWritable : which corresponds to the Java long.

Text : represents the Java String.

intWritable : represents the Java integer.

Map Method provides an instance called 'Context' to write the output. 

Reduce function also uses the same basic parameters.(input key,input value and output key , output value) important note that the parameter type of the reduce function must be same as map function.

After the reduce function completes its job, the output is saved in a distributed file system. the data here has to split into smaller parts or pieces so that they can be hosted in each machine through YARN (Yet Another Resource Negotiator).

Conclusion:

Map Reduce Job has three important things:

1. Input data

2. Map Reduce Programs

3. Configuration information.

the jobs are divided into tasks: Map task  and Reduce task.

these tasks are scheduled using YARN and run on nodes in the cluster. if a task fails, it will be automatically rescheduled to run on a different node.

here the Hadoop divides the input to a map reduce job into fixed size pieces called input splits or just splits. Hadoop create one map task for each split, which returns the user-defined map function for each record in the split. 

Flow:

1. Input is divided into small pieces  (say same size) and split to map reduce job.

2. Hadoop creates map tasks for each input split such that parallel processing is done.

3. This output is sent to reducer function where it reduces the data.

4. This all is done in nodes

 Data Locality Optimization:

Hadoop does its best to run the map task on a node hwre the input data resides in the HDFS, becaus it doesn't use valuable cluster bandwidth.

Split size should be equal to block size(to remove dominance and overhead).

Some Points To Remember:

Map tasks writes their output to the local disks not to HDFS. Because the output is intermediate output.

This intermediate output is the input for the reduce task which produces the final output. After this the final output is stored in the HDFS files. so the output of the reduce is stored in the HDFS for reliability. For each reduce task of HDFS block of the reduced output , the first replica is stored on the local node, wit the other replicas stored on the off track nodes for reliability.

Map Reduce can be a single reduce task.

it can be a multiple reduce task.

it can be a no reduce task.

Combiner Functions:

Map task produces outputs based on the splits of the input. but to feed the output as the input to the reduce function , the combiner function feeds the outputs of the Map functions as input to the reduce function.