Hadoop Shuffle

MapReduce - Shuffle

Sorting the results of Map and transferring them to Reduce for processing Map results are not stored directly to the hard disk, but use the cache to do some pre-sorting processing Map will call Comemer, compression, partitioning by key, sorting, etc., minimize the size of the results Each Map will notify Task when it is finished, and then Reduce can process it

Map end

When the Map program starts producing results, it doesn't write directly to the file, but uses the cache to do some sorting pre-processing

Each Map task has a circular memory buffer (the default 100MB), and when the cached content reaches 80%, the background thread begins to write the content to the file, at which point the Map task can continue to output the results, but if the buffer is full, the Map task will have to wait

Write a file using the round-robin method. B efore writing to the file, partition the data according to Reduce. For each partition, the key is sorted in memory, and if a Protocol is configured, It is sorted and executed (combine can then reduce the amount of data written to files and transferred)

Each time the result reaches the threshold of the buffer, a file is created, and at the end of the Map, a large number of files may be generated. T hese files are merged and sorted before map is complete. If the number of files exceeds 3, the merge will run Comemer again (1, 2 files are not necessary)

If compression is configured, the files that are eventually written are compressed first, reducing the amount of data written and transferred

Once the Map is complete, notify task manager, at which point Reduce can begin copying the resulting data

Reduce end

Map's result files are stored on the local hard drive of the machine running the Map task

If map results are small, they are placed directly in memory, otherwise written to the file

At the same time, the background thread merges and sorts these files into a larger file (if the files are compressed, they need to be unzipped first)

When all Map results are copied and merged, the Reduce method is called

The Results are written to HDFS

Tune in

The general principle is to allocate as much memory as possible to shuffle, but only if you ensure that map and Reduce tasks have enough memory

For Map, the main thing is to avoid writing files to disk, for example, using Comemer to increase the value of io.sort.mb

For Reduce, it's about saving map results as much in memory as possible, and avoiding writing intermediate results to disk. By default, all memory is allocated to the Reduce method, which can be set to 0 and mapred.job.reduce.input.buffer.percent to 1.0 if the Reduce method does not consume much memory

The number of writes to disk can be monitored through Spilled records counter in task monitoring, but this value includes map and reduce

For the IO side, map results can be compressed while increasing the buffer size (io.file.buffer.size, default 4kb)

Configuration