Coding With Fun
Home Docker Django Node.js Articles FAQ

Posts about Hadoop

Hadoop about

May 26, 2021 10:00 0 Comment Hadoop

Hadoop, Hadoop, Paving, Big data, Hadoop, The basic principles of Hadoop's components, processing processes and key knowledge points are recorded, including HDFS, YARN, MapReduce, etc.

Introduction to Hadoop

May 26, 2021 10:00 0 Comment Hadoop

Hadoop - Introduction, Hadoop - Introduction, For, Not suitable, Hadoop architecture, Hadoop - Introduction, Hadoop can be operated on general commercial servers, with high fault tolerance, high reliability, high scalability and so on,

Hadoop HDFS

May 26, 2021 10:00 0 Comment Hadoop

Hadoop - HDFS, Hadoop - HDFS, Brief introduction, Architecture, Hadoop - HDFS, Brief introduction, Hadoop Distributed File System, Distributed File System, Architecture, Block blocks;, Basic storage units, the gene

Hadoop writes the file

May 26, 2021 10:00 0 Comment Hadoop

HDFS - Write a file, HDFS - Write a file, HDFS - Write a file, 1. The client writes the file to the HDFS Client file on the local disk, 2. When the temporary file size reaches a block size, HD

Hadoop reads the file

May 26, 2021 10:00 0 Comment Hadoop

HDFS - Read the file, HDFS - Read the file, HDFS - Read the file, The client sends a read request to NameNode, NameNode returns all the locks of the files and the DataNodes in which they are loc

Hadoop reliability

May 26, 2021 10:00 0 Comment Hadoop

HDFS - Reliability, HDFS - Reliability, 1. Redundant copy policy, 2. Rack strategy, 3. Heartbeat mechanism, 4. Safe mode, 5. The effect is the same, 6. Recycle bin, 7. Metadata protection, HDFS - Reliability, There are a few key points to hdFS reliability:, Redundant replica policy, Rack strategy, Heartbeat mechanism, Safe mode, The effe

Hadoop YARN

May 26, 2021 11:00 0 Comment Hadoop

Hadoop Hadoop - YARN, Hadoop Hadoop - YARN, The old MapReduce architecture, YARN architecture, Architectural comparison, YARN basic process, Hadoop, Hadoop - YARN, The old MapReduce architecture, JobTracker: Responsible, for resource management, tracking resource consumption and availabilit

Hadoop ResourceManager

May 26, 2021 11:00 0 Comment Hadoop

YARN - ResourceManager, YARN - ResourceManager, Resource management, Task scheduling, The internal structure, YARN - ResourceManager, Responsible for global resource management and task scheduling, the entire cluster as a computing resource pool, only focus on

Hadoop NodeManager

May 26, 2021 11:00 0 Comment Hadoop

YARN - NodeManager, YARN - NodeManager, The internal structure, YARN - NodeManager, Container management under node nodes, Register with ResourceManager at startup and send a heartbeat message on a timely day, wait

Hadoop ApplicationMaster

May 26, 2021 11:00 0 Comment Hadoop

YARN - ApplicationMaster, YARN - ApplicationMaster, YARN - ApplicationMaster, Resource management and task monitoring for individual jobs, Description of the specific functions:, To calculate the applic

Hadoop Failover

May 26, 2021 11:00 0 Comment Hadoop

YARN - Failover, YARN - Failover, The type of failure, Failed processing, YARN - Failover, The type of failure, Program issues, The process crashed, Hardware issues, Failed processing, The task failed, Run-time exceptions or

Hadoop MapReduce

May 26, 2021 11:00 0 Comment Hadoop

Hadoop - MapReduce, Hadoop - MapReduce, Brief introduction, Pattern, Basic processes, Detailed process, The process under the multi-node, The main process, Hadoop - MapReduce, Brief introduction, A distributed computing method specifies a Map (insulates the value of, A function that maps a set of key valu

Hadoop reads the data

May 26, 2021 11:00 0 Comment Hadoop

MapReduce - Read data, MapReduce - Read data, InputFormat, InputSplit, RecordReader, Problem, MapReduce - Read data, The type of data read is determined by InputFormat, which is then split into One InputSplit, each InputSplit corresponds to a M

Hadoop Mapper

May 26, 2021 11:00 0 Comment Hadoop

MapReduce - Mapper, MapReduce - Mapper, MapReduce - Mapper, The main thing is to read each Key of InputSplit, value pair and process, public class Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT> {,

Hadoop Shuffle

May 26, 2021 12:00 0 Comment Hadoop

MapReduce - Shuffle, MapReduce - Shuffle, Map end, Reduce end, Tune in, Configuration, MapReduce - Shuffle, Sorting the results of Map and transferring them to Reduce for processing Map results are not stored directly to the hard disk, b