Unit 1: Introduction to Hadoop - Subjective Questions

Big Data refers to datasets whose size, complexity, and rate of generation exceed the capabilities of traditional database and data-processing systems.

Traditional systems may be inadequate because:

• Limited scalability: They usually scale vertically by adding resources to one machine, which is expensive and has physical limits.
• Rigid data models: Relational databases are primarily designed for structured data and may struggle with semi-structured and unstructured data.
• Processing constraints: A single server may require excessive time to process very large datasets.
• High data velocity: Traditional batch-oriented tools may not ingest rapidly generated data efficiently.
• Fault-tolerance limitations: Failures during large, long-running jobs may require processing to restart.

Big Data platforms address these limitations through distributed storage, parallel processing, horizontal scalability, and fault tolerance.

Major sources of Big Data include:

• Social media: Posts, comments, images, videos, reactions, and user interactions from social platforms.
• Machines and sensors: Data generated by IoT devices, industrial machines, smart meters, satellites, and environmental sensors.
• Business transactions: Sales records, banking transactions, invoices, customer orders, and payment information.
• Web and mobile applications: Clickstreams, search queries, application logs, location data, and browsing behavior.
• Scientific research: Genomic sequences, astronomical observations, medical imaging, and climate simulations.
• Public administration: Census records, transport information, healthcare records, and public-service data.

These sources continuously produce data in different formats and at different speeds, contributing to the scale and complexity of Big Data.

Key distinction: Structured data is easiest to query using conventional relational systems. Semi-structured data retains some organization through metadata, while unstructured data generally requires specialized tools for storage, indexing, and analysis.

The five V's describe the principal characteristics of Big Data:

• Volume: The enormous quantity of data generated and stored. It creates a need for scalable, distributed storage.
• Velocity: The speed at which data is generated, transmitted, and processed. High velocity may require rapid ingestion and near-real-time analysis.
• Variety: The diversity of data formats, including structured, semi-structured, and unstructured data. It requires flexible storage and processing methods.
• Veracity: The reliability, accuracy, and quality of data. Poor-quality or uncertain data can produce misleading conclusions.
• Value: The useful information and measurable benefit obtained from data. Data becomes meaningful only when analysis supports decisions, optimization, or innovation.

Together, these characteristics explain why Big Data requires technologies beyond conventional data-management systems.

Volume refers to the total amount of data, whereas Velocity refers to the rate at which data is generated, received, and processed.

Effect of Volume:

• Requires scalable and distributed storage.
• Encourages horizontal expansion by adding commodity machines.
• Requires parallel algorithms to process large datasets within acceptable time.
• Increases the importance of compression, partitioning, and replication.

Effect of Velocity:

• Requires high-throughput data-ingestion mechanisms.
• May demand low-latency stream or micro-batch processing.
• Requires systems capable of handling sudden arrival-rate spikes.
• Makes timely processing important because delayed data may lose value.

Thus, volume primarily affects capacity and processing scale, while velocity primarily affects throughput and response time.

Veracity represents the accuracy, consistency, reliability, and trustworthiness of data. Big Data may contain missing values, duplicate records, measurement errors, conflicting information, or biased samples. Data cleaning, validation, deduplication, and provenance tracking improve veracity.

Value represents the useful outcomes obtained from data, such as better decisions, reduced costs, improved services, risk detection, or new products.

Their influence on analytics is significant:

• Low-veracity data can generate inaccurate models and misleading reports.
• High-quality data does not automatically create value unless it addresses a useful problem.
• Organizations must compare the cost of collecting, storing, and analyzing data with the expected benefit.
• Effective analytics converts trustworthy data into actionable knowledge.

Therefore, veracity determines whether conclusions can be trusted, while value determines whether the analysis is worthwhile.

Apache Hadoop is an open-source framework for the distributed storage and parallel processing of very large datasets across clusters of computers.

It was designed to solve the following problems:

• Storage of massive datasets: HDFS divides large files into blocks and distributes them across multiple machines.
• Scalable processing: Hadoop executes computation in parallel on cluster nodes.
• Hardware failures: Data replication and task re-execution allow the system to continue operating when nodes fail.
• High infrastructure cost: Hadoop can run on clusters built from commodity hardware.
• Data diversity: It can store structured, semi-structured, and unstructured data without requiring a rigid relational schema.
• Data movement overhead: Hadoop attempts to move computation close to the nodes where the required data is stored.

Hadoop is especially suitable for high-throughput, batch-oriented processing of large datasets.

Hadoop has four major components:

• Hadoop Common: Supplies shared libraries, utilities, configuration facilities, and scripts.
• HDFS: Stores large files as distributed blocks with replication.
• YARN: Allocates cluster resources and schedules applications.
• MapReduce: Processes data in parallel using map and reduce stages.

A typical processing sequence is:

1. A large input file is divided into blocks and stored across HDFS DataNodes.
2. The HDFS NameNode maintains metadata about file names, blocks, permissions, and block locations.
3. A client submits a processing application to YARN.
4. The YARN ResourceManager allocates resources, while NodeManagers launch and monitor containers on worker nodes.
5. Map tasks process input partitions, preferably near the corresponding HDFS blocks.
6. Intermediate key-value records are grouped and transferred to reducers during the shuffle and sort phase.
7. Reduce tasks aggregate or transform grouped records.
8. Final output is written back to HDFS.

This cooperation provides distributed storage, parallel execution, resource management, data locality, and fault recovery.

Hadoop Common is the collection of shared libraries and utilities required by the other Hadoop modules.

Its responsibilities include:

• Providing configuration APIs and processing configuration files such as core-site.xml.
• Supplying file-system abstractions used to access HDFS and other supported storage systems.
• Providing input/output, serialization, remote procedure call, security, and networking utilities.
• Supplying scripts for starting, stopping, configuring, and administering Hadoop services.
• Providing common Java classes and dependencies used by HDFS, YARN, and MapReduce.

Hadoop Common is not primarily a storage or processing engine. Instead, it acts as the shared foundation that enables Hadoop's major components to operate consistently.

Hadoop Distributed File System (HDFS) is Hadoop's distributed storage system. It is designed to store very large files across a cluster and provide high-throughput access to their contents.

HDFS follows a master-worker architecture:

• NameNode: The master service that manages the file-system namespace and metadata. It records directories, file permissions, file-to-block mappings, and block locations. It also coordinates operations such as opening, closing, renaming, and deleting files.
• DataNodes: Worker services that store the actual data blocks on local disks. They serve client read and write requests and perform block creation, deletion, and replication when instructed.

DataNodes periodically send heartbeats and block reports to the NameNode. Heartbeats indicate that a DataNode is active, while block reports list the blocks stored by that node. This design separates metadata management from actual data storage.

HDFS divides a large file into fixed-size blocks and distributes those blocks across DataNodes. A typical block size may be 128 MB, although it is configurable.

If a file has size and the HDFS block size is , the approximate number of blocks is:

Each block is stored in multiple copies according to a configurable replication factor, commonly 3.

Scalability:

• Different blocks can be stored and processed on different machines.
• Storage capacity increases by adding DataNodes.
• Multiple clients or tasks can access different blocks in parallel.

Fault tolerance:

• If one DataNode fails, another replica can serve the same block.
• The NameNode detects failed nodes through missing heartbeats.
• Under-replicated blocks are copied to healthy DataNodes to restore the required replication level.

The trade-off is additional storage use, but replication significantly improves data availability and reliability.

HDFS read process:

1. The client requests file metadata and block locations from the NameNode.
2. The NameNode returns a list of DataNodes containing replicas of each block.
3. The client connects directly to a suitable DataNode, usually the nearest one.
4. The client reads block data and switches to other DataNodes as required.
5. If a DataNode fails, the client attempts to read another replica.

HDFS write process:

1. The client requests the NameNode to create a file.
2. The NameNode checks permissions and confirms that the file does not already exist.
3. For each block, the NameNode selects a pipeline of DataNodes according to the replication policy.
4. The client sends data packets to the first DataNode, which forwards them through the pipeline.
5. Acknowledgements travel back through the pipeline to the client.
6. When writing is complete, the client closes the file and the NameNode finalizes its metadata.

The NameNode manages metadata, but file data flows directly between clients and DataNodes.

MapReduce is Hadoop's programming model for processing large datasets in parallel across a cluster.

For a word-count application:

• Map phase: Each mapper reads part of the input and emits a key-value pair (word, 1) for every word. For example, big data big produces (big, 1), (data, 1), and (big, 1).
• Shuffle and Sort phase: Hadoop transfers intermediate records, groups identical keys, and sorts them. The example becomes (big, [1, 1]) and (data, [1]).
• Reduce phase: Each reducer combines the values belonging to a key. It computes totals such as (big, 2) and (data, 1).

Formally, the mapper and reducer can be represented as:

The framework handles input partitioning, task scheduling, intermediate-data transfer, and failed-task re-execution.

Yet Another Resource Negotiator (YARN) is Hadoop's cluster resource-management and application-scheduling layer. It separates resource management from the MapReduce processing model, allowing different distributed applications to share a Hadoop cluster.

Its principal components are:

• ResourceManager: A cluster-wide master that manages available resources and schedules applications.
• NodeManager: Runs on each worker node, launches containers, monitors resource usage, and reports node status.
• ApplicationMaster: Created for each application to negotiate resources and coordinate that application's tasks.
• Container: A bundle of allocated resources, such as memory and virtual CPU capacity, in which a task runs.

When an application is submitted, YARN starts an ApplicationMaster. The ApplicationMaster requests containers from the ResourceManager and asks appropriate NodeManagers to launch tasks. This architecture improves cluster utilization, scalability, and support for multiple processing frameworks.

Their relationship can be summarized as follows:

• HDFS answers where and how data is stored.
• YARN answers where applications can obtain computing resources.
• MapReduce answers how a batch computation is divided and executed.

For example, a MapReduce job reads input from HDFS, obtains execution resources through YARN, processes data in parallel, and writes its result back to HDFS.

• Horizontal scalability: Hadoop expands capacity by adding more machines to a cluster. Additional DataNodes contribute storage and processing resources, allowing the system to handle larger datasets and workloads.

• Fault tolerance: Hadoop expects hardware failures. HDFS stores replicated block copies, YARN monitors worker nodes, and processing frameworks can re-execute failed tasks. Consequently, a single node failure normally does not cause the entire job or dataset to become unavailable.

• Data locality: Hadoop attempts to schedule computation on, or close to, a node containing the required HDFS block. Moving a small program to the data generally consumes less network bandwidth than transferring a large dataset to a remote processor.

Together, these properties enable Hadoop to process large datasets economically and reliably on clusters of commodity machines.

Standalone mode has the simplest setup but does not provide true distributed behavior. Pseudo-distributed mode exposes HDFS and YARN behavior on one host but cannot reproduce real multi-node capacity or failure conditions. Fully distributed mode provides genuine horizontal scalability and distributed fault tolerance, but it requires more configuration, administration, networking, and security.

Major prerequisites include:

• Supported operating system: Hadoop is commonly deployed on Linux because its scripts and administration tools integrate well with Unix-like environments.
• Java Development Kit: Hadoop services execute on the Java Virtual Machine, so a compatible JDK must be installed and JAVA_HOME must be configured.
• SSH: Hadoop scripts use SSH to start and manage services, particularly across cluster nodes. Passwordless SSH is commonly configured for administrative convenience.
• Sufficient hardware resources: Adequate memory, CPU capacity, and disk space are needed for Java processes, HDFS blocks, logs, and temporary data.
• Network and hostname configuration: Nodes must resolve and communicate with one another reliably in a distributed cluster.
• User accounts and permissions: A dedicated Hadoop user is commonly used to control ownership and limit privileges.
• Hadoop distribution: A compatible Apache Hadoop release must be downloaded and extracted before configuration.

Version compatibility among Hadoop, Java, and the operating system must be verified before installation.

A typical pseudo-distributed installation includes these steps:

1. Install a compatible JDK and verify it with java -version.
2. Create or select a Hadoop user and configure passwordless SSH access to localhost.
3. Download and extract Hadoop into an installation directory.
4. Set environment variables, including JAVA_HOME, HADOOP_HOME, and the required Hadoop bin and sbin directories in PATH.
5. Configure hadoop-env.sh so Hadoop uses the correct Java installation.
6. Configure core-site.xml with the default HDFS URI.
7. Configure hdfs-site.xml with storage directories and an appropriate replication factor for a single-node setup.
8. Configure mapred-site.xml to use YARN as the MapReduce framework.
9. Configure yarn-site.xml with the required NodeManager auxiliary service.
10. Format the NameNode once using the appropriate HDFS format command.
11. Start HDFS and YARN services using Hadoop's service scripts.
12. Create user directories in HDFS, upload test data, and run a sample job.

The NameNode should not be repeatedly formatted after data has been stored because formatting creates new file-system metadata.

A Hadoop installation can be verified through the following checks:

• Run java -version and Hadoop's version command to confirm Java and Hadoop availability.
• Use jps to check whether expected Java processes, such as the NameNode, DataNode, ResourceManager, and NodeManager, are running.
• Open the HDFS and YARN web interfaces to inspect node health, storage, applications, and resource usage.
• Execute HDFS commands to create directories, upload a file, list contents, read data, and remove test files.
• Run an included example, such as word count, and verify that output is written to HDFS.
• Inspect Hadoop log files for exceptions, failed connections, permission errors, and configuration problems.

Common troubleshooting areas include:

• Incorrect JAVA_HOME or PATH values.
• SSH authentication failures.
• Invalid XML or inconsistent configuration files.
• Hostname resolution and port conflicts.
• Missing directory permissions.
• Insufficient disk space or memory.
• Services started under the wrong user account.

Verification should test both service availability and an end-to-end storage and processing workflow.