MapReduce.How does it work? - mapreduce

Question

What is MapReduce and how does it work?
I have tried reading some links but couldn't clearly understand the concept.
Can anyone please explain it in simple terms? any help would be appreciated.

Answer 1

I am going to explain with example.
Consider You have a whether temperature data of last 100 years and you want to know the year wise highest temperature. Suppose total size of the data is 100PT. How you will cater this problem? We can not process the data in SQL like oracle, My SQL, or any sql database.
In hadoop, mainly there are two term:
Hadoop Distributed File System(HDFS)
Map-Reduce
HDFS is used to store the data in distributed environment. Therefore, HDFS will store your 100PT data in cluster. It may be 2 machines cluster or 100 machines. By default your data will be divided into 64MB chunks and stored in different machine in cluster.
Now, we move to process the data. For processing data in hadoop cluster we have Map-Reduce framework to write processing logic. We need to write the map-reduce code to find the maximum temperature.
Structure of Map-Reduce code (Just for understand, syntax is not right):
class XYZ{
static class map{
void map(){
//processing logic for mapper
}
}
static class Reduce{
void reduce(){
//processing logic for reducer
}
}
}
whatever you write in map() method, that will run by all data nodes in parallel on 64MB chunk of data and generate the output.
Now, Output of all the mapper instances will shuffle and sort. And then it pass to reduce() method as a input.
Reducer will generate final output.
In our example suppose, hadoop initiate 3 below mapper:
64 MB chunk data ->mapper 1 -> (year,temperature) (1901,45),(1902,34),(1903,44)
64 MB chunk data ->mapper 2 -> (year,temperature) (1901,55),(1902,24),(1904,44)
64 MB chunk data ->mapper 3 -> (year,temperature) (1901,65),(1902,24),(1903,46)
output of all mapper pass to the reducer.
output of all mapper -> reducer -> (1901,65),(1902,34),(1903,46),(1904,44)

Answer 2

a mapreduce has a Mapper and a Reducer.
Map is a common functional programming tool which does a single operation on multiple data. For example, if we have the array
arr = [1,2,3,4,5]
and invoke
map(arr,*2)
it will multiply each element of the array, such that the result would be:
[2,4,6,8,10]
Reduce is a bit counter-intuitive in my opinion, but it is not as complicated as one would expect.
Assume you have got the mapped array above, and would like to use a reducer on it. A reducer gets an array, a binary operator, and an initial element.
The action it does is simple. Assuming we have the mapped array above, the binary operator '+', and the initial element '0', the reducer applies the operator again and again in the following order:
0 + 2 = 2
2 + 4 = 6
6 + 6 = 12
12 + 8 = 20
20 + 10 = 30.
It actually takes the last result and the next array element, and applies the binary operator on them. In the represented case, we've got the sum of the array.