Unit 2 - Practice Quiz

HDFS (Hadoop Distributed File System) is the primary storage system of the Hadoop architecture, designed to store large files across multiple machines.

The original Hadoop framework consists of two core layers: HDFS for storage and MapReduce for processing.

Hadoop is designed for batch processing, meaning it efficiently processes massive amounts of data in the background rather than providing real-time responses.

HDFS stands for Hadoop Distributed File System, which is the distributed file system that provides high-throughput access to application data.

In Hadoop 2.x and later versions, the default block size in HDFS is 128 MB. This large size helps minimize disk seek time.

HDFS achieves reliability and fault tolerance by replicating data blocks (usually 3 times by default) across different nodes in the cluster.

HDFS is designed for a 'write-once, read-many' model. A file once created, written, and closed need not be changed, which simplifies data coherency.

The Reduce function takes the intermediate output from the Map phase and aggregates or summarizes it to produce the final output.

The MapReduce framework operates exclusively on <key, value> pairs, meaning the map function outputs intermediate key-value pairs which are then processed by the reduce function.

The 'Shuffle and Sort' phase collects the output from the mappers, sorts them by key, and transfers them to the reducers.

An InputSplit is a logical division of the input data. Each Map task processes exactly one InputSplit.

The RecordReader reads data from an InputSplit and converts it into key-value pairs so the Mapper can process them.

The key-value pairs generated by the Map function are called Intermediate Data. This data is temporarily stored before being sent to the Reducer.

The NameNode acts as the master server in HDFS. It manages the file system namespace and stores all metadata, such as file names, permissions, and block locations.

DataNodes are the worker nodes in HDFS. Their primary role is to store and retrieve the actual data blocks when told to by the NameNode or the client.

In early versions of Hadoop without High Availability, the NameNode was a single point of failure. If it failed, the metadata was unavailable, making the entire file system inaccessible.

The JobTracker in MRv1 is the master service responsible for resource management, scheduling MapReduce jobs, and tracking their progress.

To achieve data locality, a TaskTracker usually runs on the same physical machine as a DataNode, allowing it to process data that is stored locally.

The standard Word Count MapReduce program outputs a text file containing key-value pairs, where the key is a unique word and the value is the number of times it appeared.

The hadoop jar command is used to run a MapReduce job packaged in a JAR file on a Hadoop cluster.

The file is split into 3 logical blocks (128 MB, 128 MB, and 44 MB). Since the replication factor is 3, each of the 3 logical blocks will have 3 replicas, resulting in physical blocks stored in the cluster.

To balance fault tolerance and write bandwidth, Hadoop's default rack awareness policy places the first replica on the local rack, and the second and third replicas on two different nodes within a single remote rack. This protects against a single rack failure.

Data locality means moving the computation to the data rather than moving data to the computation. By executing map tasks on the nodes where the HDFS blocks are stored, Hadoop drastically reduces network overhead and improves performance.

The NameNode stores metadata for every file and block in memory (RAM). Even if the files are tiny (e.g., a few kilobytes), each still consumes a fixed amount of memory for metadata (roughly 150 bytes). Millions of small files can exhaust NameNode memory before disk space runs out.

The Secondary NameNode is not a high-availability backup. Its purpose is to perform checkpointing by periodically merging the edits log into the fsimage to prevent the edits file from growing too large, which would cause long NameNode startup times.

HDFS uses a replication pipeline. The client streams the data to the first DataNode, which immediately forwards it to the second DataNode, which forwards it to the third. This pipelining efficiently utilizes network bandwidth.

If the number of reducers is zero, the MapReduce job becomes a 'map-only' job. The shuffle and sort phases are skipped, and the output of the map phase is written directly to HDFS without being sorted.

A Combiner acts as a 'mini-reducer' running locally on the output of a Map task. By aggregating data locally before it is sent over the network to the Reducers, it significantly minimizes the bandwidth required during the shuffle phase.

The framework guarantees that all values for a given key are grouped together and passed to the reducer's reduce() method. Furthermore, the keys are sorted, allowing the reducer to process the data sequentially.

A Block is a physical chunk of a file stored in HDFS (e.g., 128 MB). An InputSplit is a logical representation of data created by the InputFormat, which the RecordReader parses into key-value pairs for exactly one map task. Often they align, but an InputSplit respects logical record boundaries.

In Hadoop, data passed between phases must be serializable (Writable). Furthermore, because keys must be sorted and grouped during the shuffle phase before reaching the Reducer, they must also be comparable (Comparable). Thus, keys implement WritableComparable.

The RecordReader interacts directly with the InputSplit. It reads the raw data from the data source and translates it into <key, value> pairs that are passed one by one to the map() function.

In classic Hadoop (MRv1), the JobTracker is the central service responsible for resource management, task scheduling, monitoring the progress of jobs, and handling failures by re-assigning tasks to available TaskTrackers.

TaskTrackers send regular heartbeats to the JobTracker. If the JobTracker misses heartbeats for a certain threshold, it assumes the TaskTracker has failed and re-assigns any running tasks from that node to other healthy nodes.

Speculative Execution is a feature where the JobTracker detects a slow-running task (a straggler) and launches a duplicate, or speculative, task on another node. Whichever task finishes first is used, and the other is killed.

To achieve data locality, slave nodes in a classic Hadoop cluster run both a DataNode daemon (for storage) and a TaskTracker daemon (for computation). This allows tasks to be processed on the same node where the data is stored.

DataNodes periodically send Block Reports to the NameNode. A Block Report contains a list of all HDFS blocks stored on that DataNode. If a disk fails, the blocks will be missing from the report, prompting the NameNode to replicate them elsewhere.

Hadoop strictly requires that the output directory for a MapReduce job does not exist prior to execution. This safety mechanism prevents a job from accidentally overwriting the output of a previous job. If it exists, the job setup fails immediately.

By default, MapReduce jobs write their final output to HDFS in the specified output directory. Each Reducer creates its own output file, typically named part-r-00000, part-r-00001, etc., depending on the number of reduce tasks.

To prevent split-brain in an HA setup, Hadoop uses 'fencing' methods. If a failover occurs, the cluster employs techniques (like shutting down the old NameNode's network port or even cutting its power via an intelligent PDU) to ensure it can no longer write to the shared storage.

When a DataNode fails in the pipeline, the pipeline is temporarily closed. Any data in the ack queue is returned to the data queue. The remaining DataNodes receive a new generation stamp so the failed node's partial block is discarded if it recovers. The write resumes on the remaining nodes; the NameNode will later asynchronously replicate the block to meet the target replication factor.

If a TaskTracker's heartbeat times out, the JobTracker assumes it is dead and reschedules its tasks. The TaskTracker, however, continues executing its tasks blindly until the network is restored. Once it reconnects, the JobTracker recognizes the stale state and sends a command to the TaskTracker to kill those obsolete tasks.

Speculative execution is designed to counter hardware slowness by launching duplicate tasks on different nodes. However, it cannot solve data skew. If a Reducer is slow because it received vastly more data, the speculative duplicate will receive the exact same skewed data and suffer the same delay, simply wasting cluster resources.

TextInputFormat uses LineRecordReader, which is designed to handle records crossing block boundaries. The Mapper assigned to a split will read past the end of its block to finish the last record. Conversely, the Mapper for the next split will skip the first partial record it encounters, knowing the previous Mapper handled it.

During startup, the NameNode loads the FsImage into memory and replays the EditLog to reach the latest state. It then flushes this updated state back to disk as a new FsImage and clears the EditLog (since its transactions are now in the FsImage). Only after this process is complete and block reports are received does it exit SafeMode.

Setting the number of Reducers to 0 creates a Map-only job. In this scenario, the shuffle, sort, and reduce phases are completely bypassed. The Mappers output their raw key-value pairs directly to HDFS. For Word Count, this means outputting (word, 1) for every occurrence without any global aggregation.

Hadoop's default block placement policy places the first replica on the local node (or a random node if the client is outside the cluster). To balance fault tolerance and write bandwidth, the second replica is placed on a node in a different remote rack, and the third replica is placed on a different node in that same remote rack. This minimizes cross-rack network traffic while ensuring survival if a single rack fails.

In HDFS, writers hold a lease managed by the NameNode. If the client cannot renew the lease (due to a partition), the soft limit (1 min) and eventually the hard limit (1 hour) will expire. Upon hard limit expiration, the NameNode forcefully revokes the lease, triggers lease recovery on the last block via the DataNodes, and closes the file, guaranteeing the file system's integrity.

For Secondary Sorting, the Partitioner must partition only by the natural key (category) so all related records go to the same Reducer. The SortComparator must sort the composite key by [category, timestamp] during the shuffle phase. Finally, the GroupingComparator must group only by the natural key (category) so the Reducer's reduce() method receives all timestamps for a category in one iterable.

A Combiner acts as a mini-reducer on the map side. Because Hadoop does not guarantee how many times a Combiner will be called (it could be zero, once, or multiple times), the function must be commutative and associative (like addition or finding a maximum). Furthermore, its input types must match its output types so it can safely be chained or bypassed.

When an HDFS client reads data, it verifies checksums. If it detects a corrupted block, it reports the corruption to the NameNode (so the NameNode can schedule block replication from a healthy replica to fix the corruption) and transparently switches to another DataNode that holds a healthy replica of the block to continue the read operation.

Delay scheduling is an optimization in schedulers (like Fair and Capacity schedulers). Instead of immediately assigning a rack-local or off-switch task when node-local data isn't available, the scheduler temporarily skips the job and waits a short time. This highly increases the probability that a node-local task will become available shortly, improving overall cluster throughput.

Map tasks write their output to a circular memory buffer (default 100 MB). When the buffer fill reaches a threshold (default 80%), a background thread spills the data to local disk. The Mapper continues producing output into the remaining 20% of the buffer. If the buffer fills completely, the map thread blocks until the spill finishes.

Not all compression codecs are splittable. Gzip creates a continuous compressed stream without synchronization markers, meaning a Mapper cannot jump to the middle of the file to start reading. Thus, a Gzip file must be processed by one Mapper. Codecs like bzip2 or LZO (with indexing) provide sync points, allowing Hadoop to split them across multiple Mappers.

The WritableComparable interface combines Writable (for serialization/deserialization) and Comparable (for sorting). compareTo() is used to sort keys during the shuffle and sort phase. readFields() is the method from the Writable interface used to deserialize the data from the binary stream (DataInput) back into the object's fields.

In MRv1, the JobTracker performed dual duties: managing cluster resources and tracking the status/lifecycle of every single task in every job. This tight coupling and heavy workload caused the JobTracker to become a CPU and memory bottleneck, severely limiting the maximum size of a Hadoop cluster (typically maxing out around 4,000 nodes). YARN solved this by splitting these roles into the ResourceManager and ApplicationMaster.

In classic MRv1, the JobTracker is a single point of failure (SPOF) without native High Availability. Job states, task progress, and scheduling metadata are kept in the JobTracker's RAM. If it crashes, all running jobs are lost and must be resubmitted. Upon restart, TaskTrackers reconnect but old jobs cannot be resurrected.

When an archive (e.g., zip, tar, tgz) is added to the DistributedCache using addCacheArchive (or the newer addArchiveToClassPath), the framework automatically copies the archive to the local disk of the TaskTracker, extracts (un-archives) it, and creates a symlink in the working directory of the task, allowing local file I/O access to the extracted contents.

Data locality involves placing the computation near the data. A 'Node-local' map task runs on the exact same physical machine that holds the HDFS block, avoiding network transit. A 'Rack-local' task runs on a different machine than the data block, but on the same rack, meaning data only traverses the top-of-rack switch, which is slower than node-local but faster than off-rack.

SequenceFile supports NONE, RECORD, and BLOCK compression. RECORD compresses only the values. BLOCK compression aggregates multiple records (both keys and values) into blocks and compresses the block as a whole. Because it applies compression over a larger data payload, it achieves significantly better compression ratios. It also writes sync markers between blocks, ensuring the file remains splittable for MapReduce.