Unit 2: Hadoop Architecture - Subjective Questions

Hadoop is an open-source framework designed to store and process very large datasets across clusters of commodity computers.

Major characteristics of Hadoop architecture include:

• Distributed storage: Hadoop stores data across multiple machines using the Hadoop Distributed File System (HDFS).
• Distributed processing: Data is processed in parallel using the MapReduce programming model.
• Master-worker architecture: Master services coordinate storage and processing, while worker services perform the actual operations.
• Scalability: Storage and processing capacity can be increased by adding more machines to the cluster.
• Fault tolerance: Hadoop automatically handles hardware failures through replication and task re-execution.
• Data locality: Computation is moved close to the machine containing the required data, reducing network traffic.
• Commodity hardware support: Hadoop does not require expensive specialized hardware.

In Hadoop 1.x, the principal components are the NameNode, DataNodes, JobTracker, and TaskTrackers.

Hadoop 1.x follows a master-worker architecture consisting of two major layers:

1. Storage layer: HDFS

• The NameNode is the storage master.
• It maintains filesystem metadata, including file names, permissions, block locations, and directory structure.
• DataNodes are storage workers that hold the actual data blocks.
• DataNodes send heartbeats and block reports to the NameNode.

2. Processing layer: MapReduce

• The JobTracker is the processing master.
• It accepts jobs, divides them into tasks, schedules tasks, and monitors execution.
• TaskTrackers are processing workers that execute map and reduce tasks.
• TaskTrackers periodically report their status to the JobTracker.

A typical worker machine may run both a DataNode and a TaskTracker. This arrangement supports data locality, because the JobTracker can schedule a map task on or near the machine storing its input block.

HDFS, or the Hadoop Distributed File System, is Hadoop's distributed storage system. It stores large files across multiple machines and presents them as part of a single logical filesystem.

Its main design goals are:

• Fault tolerance: File blocks are replicated across different DataNodes.
• High-throughput access: HDFS is optimized for streaming large quantities of data.
• Scalability: The filesystem can grow by adding DataNodes.
• Large-file support: It is designed for files ranging from gigabytes to terabytes or more.
• Data locality: Processing can be scheduled near the stored blocks.
• Commodity hardware operation: Machine failures are treated as normal events.

HDFS is suitable for:

• Batch analytics
• Log processing
• Data warehousing
• Large-scale indexing
• Machine-learning data preparation

It is less suitable for applications requiring low-latency random access, frequent in-place updates, or efficient storage of a very large number of small files.

When a file is written to HDFS, it is divided into fixed-size units called blocks. These blocks are stored on different DataNodes.

For example, suppose:

• File size = MB
• HDFS block size = MB
• Replication factor =

The number of logical blocks is:

The blocks contain approximately MB, MB, and MB. With a replication factor of , each block has three copies, so HDFS maintains nine block replicas in total.

The process includes:

• The client asks the NameNode for suitable DataNodes.
• The client sends block data through a replication pipeline.
• Replicas are placed on multiple DataNodes, often across racks.
• The NameNode records the mapping between files, blocks, and DataNodes.

Replication provides fault tolerance. If one DataNode fails, a client can read another replica, and the NameNode can arrange creation of a replacement replica.

The HDFS write process proceeds as follows:

1. The client sends a create request to the NameNode.
2. The NameNode checks permissions, validates the path, and confirms that the file does not already exist.
3. The NameNode selects DataNodes for the replicas of the first block.
4. The client divides data into packets and sends them to the first DataNode.
5. The first DataNode forwards each packet to the second DataNode, which forwards it to the third. This is called a replication pipeline.
6. Acknowledgements travel back through the pipeline in reverse order.
7. When a block is full, the client requests a new set of DataNodes for the next block.
8. The process continues until all data has been written.
9. The client closes the file, and the NameNode marks the file as complete.

If a DataNode fails during writing, the pipeline is reconstructed using healthy nodes. The NameNode later ensures that every block reaches the required replication factor.

The HDFS read process involves the following steps:

1. The client sends an open request to the NameNode.
2. The NameNode returns metadata containing the file's block sequence and the DataNodes holding replicas of each block.
3. The replicas are generally ordered according to network proximity to the client.
4. The client directly contacts the nearest suitable DataNode and reads the first block.
5. The client then connects to a DataNode containing the next block.
6. This continues until the entire file has been read.
7. The client verifies checksums to detect corrupted data.

The NameNode does not carry the actual file data; it only supplies metadata. Direct communication between the client and DataNodes prevents the NameNode from becoming a data-transfer bottleneck.

If reading from one replica fails, the client selects another replica and reports the failed or corrupted copy so that Hadoop can restore the required replication level.

The NameNode is the master service of HDFS. It manages the filesystem namespace and metadata rather than storing ordinary file contents.

Its responsibilities include:

• Maintaining file and directory names
• Recording permissions and ownership
• Mapping files to HDFS blocks
• Tracking the DataNodes that contain block replicas
• Processing file operations such as create, open, rename, and delete
• Monitoring DataNodes through heartbeats and block reports
• Initiating replication when blocks become under-replicated
• Managing block deletion and re-replication

Important metadata structures include the FsImage, which stores a checkpoint of the filesystem namespace, and the EditLog, which records subsequent namespace changes.

This metadata is critical because DataNodes store blocks using block identifiers and do not independently maintain the complete file-to-block mapping. If the NameNode metadata is unavailable or permanently lost without recovery facilities, the cluster may be unable to reconstruct the logical filesystem even if block data remains on the DataNodes.

A DataNode is an HDFS worker service responsible for storing and serving actual data blocks on a cluster machine.

Its main responsibilities are:

• Storing HDFS block replicas on local disks
• Serving client read and write requests
• Creating, deleting, and replicating blocks when instructed by the NameNode
• Maintaining checksums for detecting block corruption
• Participating in replication pipelines

A DataNode communicates with the NameNode using:

• Heartbeats: Periodic messages indicating that the DataNode is alive and operational.
• Block reports: Lists of all block replicas currently stored by the DataNode.

If the NameNode stops receiving heartbeats from a DataNode for a configured period, it marks that node as unavailable. The replicas on that node are treated as missing, and the NameNode schedules re-replication from surviving copies. DataNodes do not make decisions about the filesystem namespace; they follow instructions issued by the NameNode.

The NameNode and DataNode have different roles in HDFS:

Thus, the NameNode manages the logical organization of HDFS, whereas DataNodes provide the physical storage of file blocks.

The MapReduce paradigm is a distributed programming model for processing large datasets as key-value pairs.

Its major stages are:

1. Input splitting: Input data is divided into logical InputSplits.
2. Record reading: A RecordReader converts input records into key-value pairs.
3. Map phase: Each mapper processes an input pair and emits intermediate key-value pairs.
4. Combiner phase: An optional combiner performs local aggregation of mapper output.
5. Partitioning: A partitioner assigns each intermediate key to a reducer.
6. Shuffle: Intermediate records are transferred from mapper nodes to reducer nodes.
7. Sort and group: Records are sorted by key, and values belonging to the same key are grouped.
8. Reduce phase: Each reducer processes a key and its collection of values.
9. Output phase: An OutputFormat writes reducer results, normally to HDFS.

Conceptually:

This model supports parallel execution, fault recovery, and scalable batch processing.

The terms have the following meanings:

• Job: A complete MapReduce computation submitted by a client. It includes program code, configuration, input paths, and output paths.
• Task: An individual unit of execution belonging to a job. A task is either a map task or a reduce task.
• InputSplit: A logical description of a portion of input assigned to one mapper. It normally contains location and length information rather than the data itself.
• Mapper: A function or process that reads input key-value pairs and emits zero or more intermediate key-value pairs.
• Reducer: A function or process that receives an intermediate key together with all values associated with it and emits final output records.

A job normally contains multiple map tasks and one or more reduce tasks. InputSplits determine the number of map tasks, while the number of reducers is generally specified through job configuration.

These concepts belong to different abstraction levels:

An InputSplit often corresponds closely to an HDFS block because this improves data locality, but the two are not identical. An InputSplit may span block boundaries, and certain input formats may create splits based on logical structure.

Within an InputSplit, the RecordReader identifies individual records. For a text file, each line may become a record where the key is the byte offset and the value is the line's text. Therefore, one InputSplit normally contains many records.

Partitioning, shuffle, and sort connect the map phase to the reduce phase.

• Partitioning: Determines which reducer receives each intermediate key. A common default rule is:

where is the number of reducers. All occurrences of the same key must be sent to the same reducer.

• Shuffle: Transfers intermediate mapper output across the network to the appropriate reducers. Each reducer fetches its assigned partitions from all mappers.

• Sort and grouping: Sorts intermediate records by key and groups all values belonging to an identical key. The reducer consequently receives data in the form .

These operations are essential because mapper results may be distributed across many machines. They ensure that all values for a particular key are collected at one reducer, allowing correct aggregation. However, shuffle is expensive because it involves disk input/output, serialization, sorting, and network transfer.

A combiner is an optional local aggregation function applied to mapper output before data is transferred to reducers. It can reduce the volume of intermediate data sent across the network.

In word count, a mapper may produce:

A combiner can locally aggregate these as:

This decreases shuffle traffic and may improve performance.

Important limitations are:

• Hadoop does not guarantee how many times a combiner will run.
• It may run zero, one, or multiple times.
• The final result must remain correct regardless of combiner execution.
• The operation should generally be associative and commutative.
• A reducer cannot automatically be used as a combiner when its input and output types or semantics differ.

Summation is suitable for a combiner, whereas directly calculating a simple average is unsafe unless both sum and count are preserved.

The JobTracker is the master service responsible for coordinating MapReduce jobs in Hadoop 1.x.

Its responsibilities include:

• Accepting jobs submitted by clients
• Reading job configuration and input information
• Creating map and reduce tasks
• Scheduling tasks on available TaskTrackers
• Attempting to place map tasks near their input data
• Monitoring task progress through status reports and heartbeats
• Detecting failed or unresponsive TaskTrackers
• Re-executing failed tasks on other machines
• Maintaining job status, counters, and diagnostic information
• Reporting job completion or failure to the client

Because one JobTracker manages both cluster resources and job execution, it can become a scalability bottleneck and a single point of failure in traditional Hadoop 1.x. Hadoop 2 introduced YARN, which separates resource management from application coordination.

A TaskTracker is a worker service in the Hadoop 1.x MapReduce architecture. It runs map and reduce tasks assigned by the JobTracker.

Its major functions are:

• Managing a configured number of map and reduce execution slots
• Receiving task assignments from the JobTracker
• Launching task attempts, commonly in separate Java Virtual Machines
• Monitoring task execution and resource use
• Managing intermediate mapper output on local storage
• Reporting task progress, completion, and failure
• Sending periodic heartbeats to the JobTracker

The heartbeat informs the JobTracker that the TaskTracker is alive and indicates whether execution slots are available. The JobTracker can return new task assignments in response.

If a TaskTracker fails or stops sending heartbeats, the JobTracker marks it as unavailable. Completed map output stored locally on that machine may be lost, so the relevant map tasks can be executed again on another TaskTracker.

The JobTracker and TaskTracker cooperate to execute MapReduce jobs but have distinct responsibilities:

The JobTracker provides centralized control, while TaskTrackers supply distributed execution capacity. This centralized design is straightforward but limits the scalability and availability of Hadoop 1.x.

Assume the input contains two lines:

• big data big
• data hadoop

1. Input stage

A text input format may generate records such as:

The keys are byte offsets and the values are lines.

2. Map stage

The mapper tokenizes each line and emits:

3. Optional combiner stage

Local occurrences may be summed before transfer.

4. Shuffle and sort stage

Hadoop groups values by key:

5. Reduce stage

For each key, the reducer calculates:

6. Output stage

The final output is:

• big 2
• data 2
• hadoop 1

The reducer output is written to HDFS in one or more part files.

A typical command-line word-count workflow is:

1. Create an HDFS input directory

hdfs dfs -mkdir -p /user/student/wordcount/input

2. Upload a local text file

hdfs dfs -put sample.txt /user/student/wordcount/input/

3. Verify the uploaded input

hdfs dfs -ls /user/student/wordcount/input

4. Run the word-count job

hadoop jar hadoop-mapreduce-examples.jar wordcount /user/student/wordcount/input /user/student/wordcount/output

The exact JAR path depends on the Hadoop installation.

5. List output files

hdfs dfs -ls /user/student/wordcount/output

6. Display the result

hdfs dfs -cat /user/student/wordcount/output/part-r-*

The output directory must not already exist. Before rerunning the job, it can be removed with:

hdfs dfs -rm -r /user/student/wordcount/output

The resulting part files contain each word followed by its total count.

Hadoop combines storage and processing mechanisms to provide fault tolerance and data locality.

HDFS fault tolerance:

• Each block is stored as multiple replicas on different DataNodes.
• DataNodes send heartbeats and block reports to the NameNode.
• Missing heartbeats cause a DataNode to be marked unavailable.
• Under-replicated blocks are copied from healthy replicas.
• Checksums are used to detect corrupted block data.
• Rack-aware placement can protect data against rack-level failures.

MapReduce fault tolerance:

• TaskTrackers report progress to the JobTracker.
• Failed task attempts are re-executed on another worker.
• Map output can be regenerated if the worker storing it fails.
• Slow tasks may be duplicated through speculative execution, depending on configuration.

Data locality:

• The scheduler attempts to run a mapper on the DataNode containing its input block.
• If node-local execution is unavailable, a rack-local machine may be selected.
• Moving computation to data reduces network transfer and improves throughput.

Together, these mechanisms allow Hadoop to continue processing despite common machine and disk failures.