Unit 1: Introduction to Hadoop

I. Orientation — Big Data and Distributed Computing

Big data refers to datasets whose scale, speed, diversity, or complexity exceeds the practical capabilities of traditional single-machine data-processing systems. Hadoop, created from ideas associated with Google’s distributed file and processing models and developed as an Apache project from 2006, addresses this problem by coordinating storage and computation across clusters of commodity computers.

A. Governing Principles

The unit rests on the principle that large datasets become manageable when storage and processing are distributed across multiple networked machines.

• Horizontal scaling: Capacity is increased by adding machines to a cluster rather than continually upgrading one powerful server.
• Data distribution: A large file is divided into blocks and stored across multiple computers, enabling parallel access.
• Data locality: Hadoop attempts to execute computation near the machine holding the required data, reducing network transfer.
• Fault tolerance: Data replication and task re-execution allow processing to continue when disks, processes, or machines fail.
• Parallel processing: A large job is divided into smaller tasks that can run simultaneously on separate nodes.
• Commodity hardware: Hadoop is designed for clusters of comparatively inexpensive, commonly available servers.
• Batch orientation: Traditional Hadoop is optimized for high-throughput processing of large datasets rather than millisecond-response transactional work.
• Cluster terminology:
  • Node: One physical or virtual computer in the cluster.
  • Daemon: A background service such as a NameNode or DataNode.
  • Cluster: The connected group of nodes managed as one distributed system.

II. Big Data — Data Beyond Conventional Processing

A. Introduction to Big Data

Big data is data whose size, generation rate, variety, or quality requirements demand scalable storage and distributed processing techniques.

• Relative meaning: “Big” has no permanent numerical threshold; a 10 TB dataset may be manageable in one organization but excessive in another because hardware, deadlines, and tools differ.
• Operational definition: Data becomes big when conventional databases or single-server programs cannot store, process, or analyze it within acceptable time and cost limits.
• Common sources:
  • Business systems: Sales transactions, banking records, invoices, and customer interactions.
  • Digital platforms: Search queries, clickstreams, social-media posts, images, and videos.
  • Machines and sensors: GPS devices, industrial equipment, smart meters, and Internet of Things devices.
  • Scientific systems: Satellite imagery, genomic sequences, weather observations, and particle-physics measurements.
• Scale units: Digital data is commonly measured in gigabytes, terabytes, petabytes, and exabytes; approximately (1\text{ PB}=1024\text{ TB}) when binary multiples are used.
• Analytics objective: Organizations process big data to identify patterns, make predictions, detect anomalies, personalize services, or support decisions.
• Processing styles:
  1. Batch processing: A bounded collection of records is processed together, as in calculating yesterday’s total sales.
  2. Stream processing: Events are analyzed as they arrive, as in detecting suspicious card transactions within seconds.
• Important distinction: Big data is not merely “a lot of data”; complexity, rapid arrival, uncertain quality, and distributed ownership can make even smaller datasets difficult to manage.
• Practical limitation: Collecting data does not automatically produce value; reliable results also require governance, security, suitable algorithms, and domain knowledge.

B. Types of Data

Data is classified mainly by how strongly its structure is defined and how easily conventional systems can organize it.

1. Structured data

   • Definition: Data follows a fixed schema of rows, columns, names, and data types.
   • Concrete forms: Relational database tables, spreadsheets, payroll records, and point-of-sale transactions.
   • Example schema: A sales table may contain sale_id INTEGER, date DATE, and amount DECIMAL.
   • Processing: SQL can directly filter, join, aggregate, and validate the records.
   • Strength: A known schema supports consistency and efficient queries.
   • Limitation: Rigid schemas are less convenient when fields change frequently.

2. Semi-structured data

   • Definition: Data lacks a fixed tabular schema but contains tags, keys, delimiters, or metadata that reveal organization.
   • Concrete forms: JSON documents, XML files, email headers, web-server logs, and key-value records.
   • Example:

{"sensorId":"S17","temperature":31.4,"unit":"C"}

• Interpretation: Keys such as sensorId provide structure, while different records may contain different fields.
• Strength: The format is flexible and suitable for exchanging data between applications.
• Limitation: Validation and querying may require parsing or schema inference.

3. Unstructured data
   • Definition: Data has no predefined machine-readable model comparable to a relational table.
   • Concrete forms: Natural-language documents, audio recordings, images, surveillance video, and scanned forms.
   • Processing: Text mining, natural-language processing, speech recognition, or computer vision is often required.
   • Important qualification: Files still possess metadata such as name, size, timestamp, and media type, even when their content is unstructured.
   • Limitation: Meaning depends heavily on context and is harder to extract accurately.

• Combined datasets: A healthcare system may combine structured patient identifiers, semi-structured device messages, and unstructured X-ray images; Hadoop can retain all three without forcing them into one relational schema.

C. V's of Big Data

The V’s describe the principal characteristics that make big-data systems difficult to design and operate.

• Volume: The amount of stored or processed data.

  • A video platform may retain petabytes of media and activity logs.
  • Distributed storage addresses volume by spreading blocks across many disks.

• Velocity: The rate at which data is generated, transmitted, and expected to be processed.

  • Thousands of sensor readings per second require rapid ingestion.
  • Traditional Hadoop MapReduce handles batch workloads; low-latency streams commonly require complementary streaming systems.

• Variety: The coexistence of structured, semi-structured, and unstructured formats.

  • A customer analysis may combine SQL transactions, JSON clickstreams, and text reviews.
  • Flexible storage avoids requiring every record to share one schema before ingestion.

• Veracity: The accuracy, completeness, consistency, and trustworthiness of data.

  • Duplicate customers, missing timestamps, faulty sensor values, and biased samples reduce analytical reliability.
  • Cleaning, validation, lineage tracking, and quality rules improve veracity.

• Value: The useful outcome obtained from collecting and analyzing data.

  • A predictive-maintenance model creates value when it reduces equipment downtime, not merely when it stores more readings.
  • Value must justify infrastructure, staffing, security, and operational costs.

• Additional V’s: Some models add variability, visualization, or volatility, but the five-V model—volume, velocity, variety, veracity, and value—is the common foundation.

• Interdependence: High volume without veracity can amplify errors, while high velocity without timely analysis can reduce value.

III. Hadoop — A Distributed Data Platform

A. Introduction to Hadoop

Apache Hadoop is an open-source framework for reliable, scalable storage and batch processing of large datasets across clusters of computers.

• Origin: Hadoop was developed by Doug Cutting and Mike Cafarella, influenced by Google publications describing the Google File System and MapReduce.
• Core model: Files are distributed through HDFS, while cluster resources and processing tasks are coordinated through YARN and MapReduce.
• Scale-out design: A cluster can grow from a few nodes to hundreds or thousands by adding storage and computing resources.
• Failure assumption: Hardware failure is treated as normal; Hadoop detects unavailable nodes, preserves replicated data, and reruns failed tasks.
• Schema-on-read: Raw files can be stored first and interpreted when processed, unlike schema-on-write databases that validate structure before insertion.
• Suitable workloads: Log analysis, indexing, data transformation, archival storage, and large-scale aggregation fit Hadoop’s batch-oriented design.
• Unsuitable workloads: HDFS is not ideal for large numbers of tiny files, frequent in-place record updates, or low-latency online transaction processing.
• Deployment modes:
  • Standalone mode: Runs as one Java process without HDFS; mainly used for testing.
  • Pseudo-distributed mode: Hadoop daemons run separately on one machine, simulating a cluster.
  • Fully distributed mode: Daemons and data are distributed across multiple networked nodes.

IV. Hadoop Architecture — Storage, Resources, and Processing

A. Components of Hadoop

Hadoop consists of four core modules that jointly provide shared utilities, distributed storage, resource management, and parallel processing.

• Hadoop Common: Supplies Java libraries, configuration handling, scripts, filesystem interfaces, and utilities required by other Hadoop modules.

• Hadoop Distributed File System (HDFS): Stores very large files as distributed blocks.

  • NameNode: Maintains filesystem metadata, including directory names, permissions, and block locations.
  • DataNode: Stores actual blocks and reports its status to the NameNode.
  • Blocks: Large blocks, commonly configured as 128 MiB, reduce metadata and support sequential access.
  • Replication: A typical default factor of three places copies on different DataNodes for fault tolerance.
  • Secondary NameNode: Periodically combines namespace image and edit files; it is a checkpointing service, not a standby replacement NameNode.

• Yet Another Resource Negotiator (YARN): Allocates cluster CPU and memory to applications.

  • ResourceManager: Makes cluster-wide scheduling and resource-allocation decisions.
  • NodeManager: Runs on each worker and monitors containers and resource use.
  • ApplicationMaster: Coordinates one application and negotiates containers.
  • Container: A bounded allocation of resources, such as memory and virtual CPU cores.

• MapReduce: Implements distributed batch computation.

  • Map phase: Converts input records into intermediate key-value pairs.
  • Shuffle and sort: Groups all intermediate values associated with each key.
  • Reduce phase: Aggregates grouped values to produce final output.
  • Concrete example: In word counting, mappers emit (word, 1) and reducers sum the values for each word.

• Component interaction: HDFS supplies blocks, YARN assigns containers, and MapReduce runs tasks close to relevant DataNodes whenever possible.

V. Hadoop Deployment — Preparing a Single-Node Environment

A. Installation of Apache Hadoop

A pseudo-distributed installation on Linux provides the main Hadoop services on one machine for learning and development.

• Prerequisites: Install a supported 64-bit Java Development Kit, SSH utilities, and a dedicated non-root user; verify Java with:

java -version
ssh -V

• Package installation: Download a verified Apache Hadoop 3.x binary archive from an official Apache mirror, then extract it:

tar -xzf hadoop-3.x.x.tar.gz
mv hadoop-3.x.x "$HOME/hadoop"

• Environment variables: Add paths to the shell profile and set the actual JDK location:

export JAVA_HOME=/path/to/jdk
export HADOOP_HOME=$HOME/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

• Java configuration: Set JAVA_HOME in etc/hadoop/hadoop-env.sh so Hadoop daemons can locate Java.
• Passwordless local SSH: Generate a key and authorize it for localhost:

ssh-keygen -t ed25519 -N "" -f "$HOME/.ssh/id_ed25519"
cat "$HOME/.ssh/id_ed25519.pub" >> "$HOME/.ssh/authorized_keys"
ssh localhost

• Core configuration: In core-site.xml, identify the default HDFS endpoint:

<property>
  <name>fs.defaultFS</name>
  <value>hdfs://localhost:9000</value>
</property>

• HDFS configuration: In hdfs-site.xml, use one replica because only one DataNode exists:

<property>
  <name>dfs.replication</name>
  <value>1</value>
</property>

• MapReduce and YARN: Set mapreduce.framework.name to yarn in mapred-site.xml; set yarn.nodemanager.aux-services to mapreduce_shuffle in yarn-site.xml.
• Filesystem initialization: Format the NameNode only for a new HDFS namespace:

hdfs namenode -format
start-dfs.sh
start-yarn.sh

• Verification: jps should list services such as NameNode, DataNode, ResourceManager, and NodeManager.
• Smoke test: Create an HDFS directory and upload a local file:

hdfs dfs -mkdir -p /user/$USER/input
hdfs dfs -put sample.txt /user/$USER/input/
hdfs dfs -ls /user/$USER/input

• Web interfaces: Common default interfaces are the NameNode at http://localhost:9870 and ResourceManager at http://localhost:8088.
• Safe shutdown:

stop-yarn.sh
stop-dfs.sh

• Operational caution: A one-node configuration demonstrates Hadoop architecture but provides neither real replication nor machine-level fault tolerance; production clusters require separate nodes, access control, monitoring, backups, and high-availability planning.