Unit5 - Subjective Questions

Apache HBase is an open-source, non-relational (NoSQL), distributed database modeled after Google's Bigtable and written in Java. It runs on top of the Hadoop Distributed File System (HDFS).

Primary Characteristics:

• Column-Oriented: It stores data in a column-oriented manner, making it highly efficient for sparse data sets.
• Scalability: It is designed to scale horizontally by adding more commodity servers.
• Strict Consistency: It provides strictly consistent reads and writes, distinguishing it from many other NoSQL stores that offer eventual consistency.
• High Availability: Built-in mechanisms for failover and fault tolerance.

Why NoSQL?
HBase does not support SQL natively, does not enforce relationships (no foreign keys), and is schema-less for columns (only column families are predefined). It is built to handle massive, sparse, unstructured, or semi-structured data.

While both HBase and RDBMS store data, their underlying architectures and use cases differ significantly.

Differences:

• Data Model: RDBMS uses a fixed schema (tables, rows, typed columns). HBase uses a schema-less column-family model where columns can be added dynamically.
• Storage Mechanism: RDBMS is typically row-oriented. HBase is column-oriented.
• Transactions: RDBMS supports complex, multi-row ACID transactions. HBase only guarantees ACID properties at the row level.
• Query Language: RDBMS uses SQL. HBase uses specialized APIs (Java, REST, Thrift) and Shell commands, though tools like Apache Phoenix can provide a SQL layer.
• Scalability: RDBMS scales vertically (requires a more powerful server). HBase scales horizontally across distributed clusters.
• Data Volume: RDBMS is suited for gigabytes to terabytes. HBase is designed for petabytes of data.

HDFS and HBase are both integral parts of the Hadoop ecosystem, but they serve different purposes.

HDFS (Hadoop Distributed File System):

• Suitable for storing large files and performing batch processing (e.g., MapReduce).
• Follows a Write-Once-Read-Many (WORM) model. Data cannot be easily updated once written.
• Sequential access; high latency for data retrieval.
• Does not support random reads/writes.

HBase:

• Built on top of HDFS to provide low-latency, random read/write access to data.
• Allows real-time querying of large datasets.
• Stores data as key-value pairs in a column-oriented format.

Why HBase is needed:
HDFS is excellent for batch processing but terrible for random access. HBase bridges this gap by utilizing HDFS as its underlying storage while providing a layer that allows fast, random lookups and updates.

The HBase data model consists of several logical components that structure how data is stored and accessed:

• Table: The highest-level container. An HBase table consists of multiple rows.
• Row Key: Every row has a unique identifier called the Row Key. Rows are lexicographically sorted by this key. Operations on a single row are atomic.
• Column Family: Data within a row is grouped into Column Families. These must be defined when the table is created. All column members of a column family have the same prefix (e.g., info:name, info:age).
• Column Qualifier: Columns within a family. They are dynamic and can be added on the fly.
• Cell: The intersection of a Row Key, Column Family, and Column Qualifier. It contains the actual value/data as an uninterpreted array of bytes.
• Timestamp: Every cell has a timestamp associated with it, allowing HBase to store multiple versions of a value.

In HBase, data is never strictly overwritten in place. Instead, HBase uses versioning driven by timestamps.

• Timestamp: When a value is written to a cell, HBase automatically assigns it a timestamp (typically the current server time). Users can also explicitly define custom timestamps.
• Versioning: A cell in HBase is uniquely identified by the tuple: (Row Key, Column Family, Column Qualifier, Timestamp). This means multiple versions of a data point can exist in the same logical cell, distinguished only by their timestamp.
• Retrieval: By default, when a user queries a cell, HBase returns the version with the highest timestamp (the most recent value). Users can specify to retrieve a specific version or all versions within a time range.
• Cleanup: HBase allows setting a VERSIONS limit per column family (e.g., keep only the last 3 versions). Older versions are purged during the compaction process.

Apache HBase has a master-slave architecture built on top of HDFS. Its core components are:

1. HMaster: The master node responsible for monitoring all RegionServer instances, managing metadata operations (create, modify, delete tables), and assigning regions to RegionServers.
2. RegionServers (HRegionServer): The worker nodes responsible for serving and managing read and write requests from clients. They manage a set of 'Regions' (horizontal slices of tables).
3. Regions (HRegion): The basic unit of scalability and load balancing. A table is divided into multiple regions based on row keys.
4. Zookeeper: A distributed coordination service that maintains server status, coordinates the HMaster election, and stores the location of the hbase:meta catalog table.
5. HDFS: The underlying distributed file system where HBase physically stores data (as HFiles and WALs).

Role of HMaster:
The HMaster is responsible for cluster administration and coordination. Its duties include:

• Region Assignment: Assigning regions to RegionServers during startup, recovery, or load balancing.
• Monitoring: Monitoring the health of RegionServers via heartbeats (usually through Zookeeper).
• DDL Operations: Handling metadata changes like creating, altering, or dropping tables.
• Load Balancing: Moving regions across RegionServers to balance the load.

Data Path:
Importantly, HMaster is NOT in the read/write path. Clients communicate directly with Zookeeper to find the hbase:meta table, then connect directly to the specific RegionServer holding the required data. This prevents the HMaster from becoming a bottleneck during heavy data I/O.

The RegionServer is the workhorse of an HBase cluster. It handles actual data storage and client requests. Its key responsibilities include:

• Serving Data: Handling client read (get, scan) and write (put, delete) requests for the regions it hosts.
• Managing Regions: Managing the lifecycle of HRegions. When a region grows too large, the RegionServer automatically splits it (Region Split).
• Managing MemStore: Storing incoming writes in an in-memory buffer (MemStore) for fast write performance.
• Flushing: Periodically flushing the contents of the MemStore to disk as an HFile.
• Compaction: Performing minor and major compactions to merge HFiles and clean up deleted/expired data.
• WAL Maintenance: Writing incoming data to the Write-Ahead Log (WAL) to ensure data durability in case the server crashes.

The write path in HBase ensures high performance while maintaining strict durability.

Step-by-Step Write Process:

1. Client Request: The client routes the write request (Put) directly to the appropriate RegionServer.
2. Write-Ahead Log (WAL): The RegionServer first writes the data to the WAL stored in HDFS. This ensures data is not lost if the RegionServer crashes before memory is flushed to disk.
3. MemStore: After the WAL write is successful, the data is written to an in-memory cache called the MemStore. There is one MemStore per Column Family per Region.
4. Acknowledgment: Once data is in the MemStore, an acknowledgment is sent back to the client. This makes writes very fast.
5. Flush: When the MemStore reaches a configured threshold (e.g., 128 MB), its contents are flushed to HDFS as a new, immutable file called an HFile.

Reading data in HBase is slightly more complex than writing because data for a single row might be spread across memory and multiple disk files.

Read Process:

1. Meta Lookup: The client queries Zookeeper to find the location of the hbase:meta table, queries it to find which RegionServer hosts the requested row key, and connects to that RegionServer.
2. BlockCache: The RegionServer first checks the BlockCache (an in-memory read cache). If the data is there, it is returned immediately.
3. MemStore: If not in BlockCache, the server checks the MemStore (which holds the most recently written, un-flushed data).
4. HFiles: If the data is still not found, the RegionServer must read from the HFiles on disk. To avoid reading all HFiles, HBase uses Bloom Filters and Block Indexes to quickly skip files that do not contain the requested row key.
5. Merge: Because different versions or columns might exist in different HFiles and the MemStore, HBase merges the results before returning them to the client.

Because HBase flushes MemStores to disk sequentially, over time, a single region may accumulate many small HFiles. Compaction is the process of merging these files.

Minor Compaction:

• Mechanism: Merges a small number of adjacent, smaller HFiles into fewer, larger HFiles.
• Goal: Reduces the number of files to improve read performance.
• Note: It does not drop deleted data or expired versions.

Major Compaction:

• Mechanism: Merges and rewrites all HFiles in a Region's Column Family into a single, large HFile.
• Goal: Cleans up the system. It physically removes cells marked with delete markers (tombstones), drops expired data (TTL), and removes excess versions.
• Impact: Highly I/O intensive. Often scheduled during off-peak hours to avoid impacting cluster performance.

Role of Zookeeper:
Apache Zookeeper is the centralized coordination service for HBase. Its functions include:

• Tracking the Active HMaster: Ensuring only one HMaster is active at a time and handling failover if it crashes.
• RegionServer Tracking: RegionServers register with Zookeeper. Zookeeper monitors their heartbeats and alerts the HMaster if a RegionServer dies.
• Routing: It stores the location of the root hbase:meta table. Clients must talk to Zookeeper first to find out where to read/write data.

If Zookeeper goes down:
The entire HBase cluster ceases to function correctly. Clients cannot locate data, RegionServers cannot report their status, and the HMaster cannot manage the cluster. Zookeeper is a critical single point of dependency, which is why it is run in an odd-numbered ensemble (e.g., 3, 5, or 7 nodes) for high availability.

Hotspotting occurs when a disproportionate amount of read or write traffic is directed at a single node (RegionServer) in the cluster, leaving other nodes idle. This usually happens due to poor Row Key design (e.g., using sequential timestamps as row keys), causing all new writes to hit the same region.

Techniques to Avoid Hotspotting:

1. Salting: Adding a random prefix (salt) to the row key. This distributes sequential writes across multiple regions. (e.g., row123 becomes A-row123, B-row124).
2. Hashing: Applying a hash function (like MD5) to the natural key and using the hash as the row key. This randomly distributes keys but makes range scans impossible.
3. Reversing the Key: If the key has a predictable prefix (like a phone number or domain), reversing the string prevents hot-spotting while maintaining some logical grouping (e.g., reversing www.google.com to com.google.www).

An HFile is the physical storage format used by HBase to store data in HDFS. It is an immutable, sorted key-value file.

Structure of an HFile:

• Data Blocks: The core of the file, containing the actual key-value pairs sorted lexicographically by row key. Block size is typically 64KB.
• Meta Blocks: Used to store metadata, such as Bloom filters, which help quickly determine if a key exists in the file.
• File Info: Contains basic information like the highest sequence ID, major compaction tags, and average key/value lengths.
• Data Index: A multi-level B+ tree like index over the data blocks. It maps the first key of every data block to its offset, allowing fast lookups without scanning the whole file.
• Trailer: A fixed-size section at the end of the file pointing to the offsets of the other blocks (File Info, Data Index, etc.).

Bloom Filters are highly space-efficient probabilistic data structures used to test whether an element is a member of a set.

Significance in HBase:

• Reducing Disk I/O: During a read operation, HBase might need to check multiple HFiles. A Bloom filter can quickly answer if a specific Row Key (or Row+Column) is definitely not in an HFile, or possibly in the HFile.
• Performance Boost: By skipping HFiles that do not contain the requested data, Bloom filters drastically reduce unnecessary disk reads.
• Types in HBase:
  • Row Bloom Filter: Checks if a row key exists in the HFile.
  • RowCol Bloom Filter: Checks if a specific row key + column qualifier combination exists.
• Trade-off: They consume some memory/disk space and have a slight false-positive rate, but they never return a false negative.

Regions are the basic elements of availability and distribution for tables in HBase. A table initially consists of one Region. As data is added, the region grows.

Region Splitting Process:

• Threshold: When a region reaches a configured size threshold (e.g., hbase.hregion.max.filesize, typically 10GB), it is split into two child regions.
• Mid-Key Calculation: The RegionServer finds the middle row key (mid-key) to divide the data into two roughly equal halves.
• Offlining: The parent region is temporarily taken offline, causing a brief pause in reads/writes for those specific rows.
• Creation of Daughters: Two new child regions (daughters) are created in HDFS. Initially, they are just reference files pointing to the parent's HFiles.
• Assignment: The HMaster is notified, and it may leave the daughter regions on the same server or assign one to a different RegionServer for load balancing.
• Compaction: Later, a major compaction physically writes the daughter data into separate HFiles, and the parent is deleted.

The HBase shell is an interactive Ruby-based command-line utility.

1. create: Creates a new table.
   • Syntax: create 'tableName', 'columnFamily1', 'columnFamily2'
   • Purpose: Defines a table and its column families.
2. put: Inserts or updates a cell value.
   • Syntax: put 'tableName', 'rowKey', 'columnFamily:columnQualifier', 'value'
   • Purpose: Writes a single piece of data to a specific coordinate.
3. get: Fetches data for a specific row.
   • Syntax: get 'tableName', 'rowKey'
   • Purpose: Retrieves all column families and their values for a single row.
4. scan: Iterates over the table data.
   • Syntax: scan 'tableName'
   • Purpose: Displays all records in a table. Can be limited using start/stop rows.
5. disable / drop: Removes a table.
   • Syntax: disable 'tableName' followed by drop 'tableName'
   • Purpose: A table must be disabled before it can be completely dropped/deleted from the cluster.

Because data is accessed and sorted exclusively by the Row Key, designing it correctly is the most critical part of HBase schema design.

Best Practices:

• Prevent Hotspotting: Avoid monotonically increasing keys (like timestamps or sequential IDs). Use salting, hashing, or reversing strings to distribute writes evenly.
• Keep it Short: Row keys are stored in every cell (along with column family and qualifier). Long row keys lead to massive overhead in storage and memory (BlockCache). Limit length while ensuring uniqueness.
• Optimize for Scans: HBase sorts keys lexicographically. If you need to retrieve related items together, design the key so related entities sort adjacently (e.g., [UserID]-[Timestamp] allows fetching a user's chronological events via a targeted scan).
• Fixed Length: When combining multiple data types into a row key, pad them to a fixed length to ensure correct lexicographical sorting (e.g., integer 2 sorts after 11 as a string, but 02 sorts before 11).

Apache HBase excels in scenarios requiring fast, random access to massive, sparse datasets.

1. Time-Series Data: Storing metrics, IoT sensor data, or financial ticker data. HBase's ability to handle high-velocity writes and its innate timestamp versioning make it perfect for tracking data over time.
2. Web Crawling and Content Serving: (e.g., Apache Nutch). HBase can store billions of webpages where the row key is the reversed URL. It easily handles variable columns for headers, content, and parsed metadata.
3. Social Media and Messaging: Storing user timelines, messages, or activity logs. Applications like Facebook Messages (historically) used HBase because it handles heavy write workloads and allows instant read access to recent message history.

HBase acts as both a source (input) and a sink (output) for Hadoop MapReduce jobs, allowing for complex batch processing over NoSQL data.

Integration Mechanism:

• TableInputFormat: This class allows a MapReduce job to read data from an HBase table. It splits the table based on its Regions, meaning one Mapper is usually spawned per Region. It passes data to the Mapper as (ImmutableBytesWritable key, Result value).
• TableMapper: An abstract class provided by HBase to simplify writing Mappers that read from HBase.
• TableOutputFormat: This class allows Reducers to write data back into an HBase table. It ensures connections to RegionServers are handled properly.
• TableReducer: An abstract class used to write output directly to an HBase table using Put or Delete objects.
• This integration enables developers to run complex analytics on petabytes of unstructured data stored in HBase using the distributed compute power of MapReduce.