Unit 2: Hadoop Architecture

I. Orientation

Apache Hadoop is an open-source framework for storing and processing very large datasets across clusters of commodity computers. Inspired by Google’s distributed file-system and MapReduce papers, Hadoop combines distributed storage through the Hadoop Distributed File System (HDFS) with parallel computation through MapReduce.

• Scale-out architecture: Capacity increases by adding machines to a cluster rather than replacing one server with a larger system.
• Data locality: Hadoop attempts to execute computation on, or near, the DataNode containing the required HDFS blocks.
• Fault tolerance: HDFS replicates blocks, while MapReduce retries failed tasks on other machines.
• Shared-nothing design: Cluster nodes generally have their own processors, memory, and disks; coordination occurs over the network.
• Master–worker organization: Master services maintain metadata and coordinate work, while worker services store blocks or execute tasks.
• Batch-processing orientation: Classic MapReduce is optimized for high-throughput scans of large datasets, not low-latency transactions.
• Hadoop-version convention: NameNode and DataNode remain HDFS components; JobTracker and TaskTracker belong specifically to Hadoop 1.x MapReduce, also called MRv1.
• Core data flow: Input files are divided into HDFS blocks, map tasks process input splits, shuffle groups intermediate records, and reduce tasks write final output.

II. Core Distributed Design — Coordinating Storage and Computation

A. Hadoop Architecture

Hadoop architecture integrates distributed storage, resource coordination, and parallel data processing within a cluster.

• Hadoop Common: Shared libraries, configuration utilities, scripts, and APIs support the other Hadoop modules.
• Storage layer: HDFS stores large files as replicated blocks distributed across DataNodes.
• Processing layer: MapReduce divides a job into map and reduce tasks that can run concurrently.
• MRv1 control structure:
  • NameNode: Manages the HDFS namespace and block metadata.
  • DataNode: Stores and serves actual HDFS blocks.
  • JobTracker: Accepts jobs, schedules tasks, and monitors execution.
  • TaskTracker: Runs map and reduce tasks on a worker node.
• Hadoop 2.x and later: YARN separates resource management from processing. The ResourceManager, NodeManagers, and per-application ApplicationMaster replace JobTracker and TaskTracker responsibilities.
• Client interaction: A client contacts the NameNode for block locations but transfers file data directly to or from DataNodes.
• Rack awareness: Hadoop uses rack information when placing replicas, reducing the risk that a rack or network-switch failure makes data unavailable.
• Strengths and limits: The design offers scalable, fault-tolerant batch processing, but many small files, frequent random updates, and millisecond-response workloads are poor fits.

III. Distributed Storage — Persistent Data Across Cluster Nodes

A. Hadoop Storage: HDFS

HDFS is a distributed file system designed for large files, streaming access, and reliable operation despite ordinary hardware failures.

• Block storage: A file is divided into large blocks, commonly configured as 128 MiB in modern Hadoop installations; the final block may be smaller.
• Replication: Each block normally has a replication factor of 3, although the value is configurable per file.
• Namespace: Directories, file names, permissions, replication factors, and file-to-block mappings are managed centrally by the NameNode.
• Write-once model: HDFS primarily supports create, append, and sequential read operations rather than arbitrary in-place modification.
• Read operation:
  1. The client requests block locations from the NameNode.
  2. The client selects a nearby replica.
  3. Data is streamed directly from the chosen DataNode.
• Write pipeline:
  1. The NameNode chooses target DataNodes.
  2. The client sends packets to the first DataNode.
  3. Each DataNode forwards packets to the next replica.
  4. Acknowledgements return through the pipeline.
• Integrity checking: Checksums detect corrupted block data; a valid replica can replace a damaged one.
• Concrete example: A 300 MiB file with a 128 MiB block size requires three logical blocks: 128 MiB, 128 MiB, and 44 MiB. With replication factor 3, HDFS maintains nine block replicas.

IV. Parallel Processing — Transforming Distributed Data

A. Hadoop MapReduce paradigm

The Hadoop MapReduce paradigm processes data through a map stage that creates intermediate records and a reduce stage that aggregates records sharing a key.

• Map function: Converts an input key–value pair into zero or more intermediate key–value pairs.

map(k1, v1) → list(k2, v2)

• k1: Input key, such as a byte offset.
• v1: Input value, such as one line of text.
• k2: Intermediate grouping key, such as a word.
• v2: Intermediate value, such as the count 1.

• Shuffle and sort: Hadoop partitions mapper output, transfers it to reducers, and sorts records so all values for one key are grouped.
• Reduce function: Combines the values associated with an intermediate key.

reduce(k2, list(v2)) → list(k3, v3)

• k3: Output key.
• v3: Aggregated output value.

• Execution sequence: Input splitting → mapping → optional combining → partitioning → shuffling → sorting → reducing → HDFS output.
• Data locality: Scheduling a mapper near its input block avoids transferring an entire block across the network.
• Fault recovery: A failed task attempt is rerun; speculative execution may duplicate unusually slow tasks and accept the first successful result.
• Limitation: Iterative algorithms may repeatedly read and write HDFS data between jobs, creating more overhead than memory-oriented engines.

B. MapReduce Terminology

MapReduce terminology identifies the units of data, execution, and coordination used during a distributed job.

• Job: The complete submitted computation, including program code, configuration, input paths, and output path.
• Input split: A logical section of input assigned to one mapper; it often corresponds closely, but not necessarily exactly, to an HDFS block.
• RecordReader: Converts bytes from an input split into records such as (offset, line).
• Mapper: Executes the map function once for each input record.
• Intermediate pair: A mapper-produced (key, value) record, such as ("data", 1).
• Combiner: An optional local aggregation step that reduces network traffic; it must not be treated as guaranteed to execute.
• Partitioner: Chooses a reducer for each key, commonly using:

partition = hash(key) mod R

• hash(key): Numeric hash of the intermediate key.
• R: Number of reducers.

• Shuffle: Network transfer of mapper partitions to their assigned reducers.
• Sort and group: Orders intermediate keys and forms a key with its iterable collection of values.
• Reducer: Processes one grouped key at a time and emits final records.
• Task attempt: One execution instance of a map or reduce task; retries create additional attempts.
• Output format: Controls how final key–value records are written, commonly as text files named part-r-00000 and similar.

V. HDFS Metadata Management — The Master Storage Service

A. Hadoop NameNode

The Hadoop NameNode manages the HDFS namespace and determines where file blocks are stored, but it does not normally carry application file data.

• Metadata responsibilities: It records file paths, ownership, permissions, replication factors, and mappings from files to block identifiers.
• Persistent state:
  • FsImage: A checkpointed snapshot of the filesystem namespace.
  • Edit log: A sequence of namespace changes made after the snapshot.
• Startup behavior: The NameNode loads the FsImage, applies edit-log operations, and reconstructs current block locations from DataNode reports.
• Client coordination: It authorizes file operations and returns suitable DataNode addresses for reads or writes.
• Health monitoring: Heartbeats show that DataNodes are alive, while block reports identify blocks stored by each DataNode.
• Replication management: Missing replicas trigger re-replication; excess replicas may be deleted according to placement policy.
• Safe mode: During startup, the NameNode temporarily restricts changes until enough blocks are reported as safely replicated.
• Availability: Hadoop high availability can use active and standby NameNodes with shared edit information and automatic failover.
• Important distinction: A Secondary NameNode periodically merges FsImage and edit-log data to create checkpoints; it is not simply a live backup NameNode.

VI. HDFS Block Storage — The Worker Storage Service

A. Hadoop DataNode

A Hadoop DataNode stores HDFS block replicas on local disks and serves them to clients or other DataNodes.

• Block operations: DataNodes create, read, transmit, replicate, and delete blocks under NameNode coordination.
• Local storage: Block contents and associated checksum metadata are maintained in DataNode storage directories.
• Heartbeats: Regular heartbeat messages inform the NameNode that the DataNode remains operational.
• Block reports: Full reports list stored blocks; incremental reports describe recent additions, deletions, or changes.
• Read service: A client retrieves data directly from a selected DataNode, normally preferring the nearest replica.
• Write service: A DataNode receives packets, stores them, forwards them through the replication pipeline, and returns acknowledgements.
• Failure handling: If heartbeats stop beyond a configured interval, the NameNode marks the node dead and schedules replacement replicas elsewhere.
• Operational constraint: Removing block files manually can violate HDFS metadata consistency; administration should use HDFS commands and procedures.

VII. MRv1 Job Coordination — Centralized Scheduling and Monitoring

A. Hadoop JobTracker

The Hadoop JobTracker is the Hadoop 1.x master service that accepts MapReduce jobs, schedules their tasks, and monitors completion.

• Job submission: The client provides the job JAR, configuration, input location, output location, and supporting resources.
• Task creation: Input splits determine the number of map tasks, while job configuration specifies the number of reducers.
• Scheduling: The JobTracker assigns tasks to available TaskTrackers, preferring nodes or racks containing the required input.
• Progress tracking: TaskTracker heartbeats report available execution slots, task status, counters, and failures.
• Recovery: Failed tasks are rescheduled; repeated failure can cause the entire job to fail.
• Scalability issue: One JobTracker handled both cluster resources and job lifecycle, making it a bottleneck and single point of failure.
• Modern replacement: YARN distributes these duties among the ResourceManager, NodeManagers, and ApplicationMaster.

VIII. MRv1 Task Execution — Worker-Side Processing

A. Hadoop TaskTracker

The Hadoop TaskTracker is the Hadoop 1.x worker daemon responsible for launching and supervising assigned map and reduce task attempts.

• Execution slots: Each TaskTracker advertises configured map and reduce slots representing its concurrent task capacity.
• Task launch: It obtains job resources, creates a working directory, and launches task attempts in separate Java Virtual Machines.
• Status reporting: Heartbeats communicate task progress, completion, failure, and free slots to the JobTracker.
• Local intermediate data: Mapper output is sorted and stored on local disk before reducers fetch their partitions.
• Failure isolation: A failed attempt can be terminated and rerun without restarting the complete job.
• Node failure: If a TaskTracker stops reporting, its incomplete tasks are assigned elsewhere; completed mapper output may also need regeneration because it was local.
• Modern replacement: In YARN, NodeManagers manage containers instead of fixed TaskTracker map and reduce slots.

IX. Command-Line Execution — Running the Standard Example

A. Word count on command line

Word count on command line demonstrates the complete Hadoop workflow by loading text into HDFS, running MapReduce, and reading distributed output.

• Prepare input: Create a local file and an HDFS input directory.

printf "big data big\nhadoop data\n" > sample.txt
hdfs dfs -mkdir -p /user/student/wc-input
hdfs dfs -put sample.txt /user/student/wc-input/

• Run the example job: The exact installation path may differ, but Hadoop distributions include a MapReduce examples JAR.

hadoop jar \
  "$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-"*.jar \
  wordcount /user/student/wc-input /user/student/wc-output

• Inspect output: Hadoop requires the output directory not to exist before job submission.

hdfs dfs -cat /user/student/wc-output/part-r-00000

• Expected records:

big     2
data    2
hadoop  1

• Processing logic: Each mapper emits (word, 1); shuffle groups equal words; each reducer calculates the sum of the grouped values.
• Rerunning the job: Delete or rename the existing output directory before using the same output path.

hdfs dfs -rm -r /user/student/wc-output

• Command roles: hdfs dfs performs filesystem operations, while hadoop jar submits the packaged MapReduce program to the cluster.