Unit 5: Introduction to Apache HBase

I. Orientation

Apache HBase is an open-source, distributed, column-family NoSQL database modeled after Google Bigtable and built for the Hadoop ecosystem. It stores very large, sparse tables across clusters and provides low-latency, random read/write access, unlike HDFS, which is optimized mainly for sequential file access.

• Governing principle: HBase partitions tables by row-key ranges and distributes those partitions, called regions, among RegionServers.
• Storage foundation: Persistent data is stored in HDFS files, while ZooKeeper coordinates distributed services and tracks important cluster locations.
• Data organization: A cell is identified by the tuple (row key, column family, column qualifier, timestamp).
• Consistency model: HBase provides strongly consistent reads and writes for operations involving a single row; multirow transactions are generally not provided.
• Schema convention: Column families are declared when the table is created, but qualifiers inside a family can be added dynamically.
• Access pattern: HBase supports API and shell operations such as put, get, scan, and delete; it is not a relational SQL database.
• Design assumption: Row-key design is central because rows are physically ordered lexicographically by their byte-array keys.
• Typical use: HBase suits time-series records, event streams, sensor data, message histories, and other large datasets requiring random access.

II. Apache HBase Setup

A. Installation of Apache HBase

Installation establishes Java, Hadoop-compatible storage, configuration files, and the HBase services required for standalone or distributed operation.

• Prerequisites: Install a Java version supported by the chosen HBase release and set JAVA_HOME; distributed installations also require a functioning HDFS cluster.
• Package setup: Download and extract an Apache HBase binary release, then define its location and commands.

tar -xzf hbase-<version>-bin.tar.gz
export HBASE_HOME=/opt/hbase
export PATH=$PATH:$HBASE_HOME/bin

• Java configuration: Set the Java path in conf/hbase-env.sh.

export JAVA_HOME=/path/to/java

• Standalone configuration: A local installation can use the local filesystem and HBase-managed ZooKeeper. In conf/hbase-site.xml, define a durable absolute path rather than a temporary directory.

<configuration>
  <property>
    <name>hbase.rootdir</name>
    <value>file:///opt/hbase-data</value>
  </property>
</configuration>

• Distributed configuration: Set hbase.cluster.distributed to true, point hbase.rootdir to HDFS, and identify ZooKeeper quorum hosts.

<property>
  <name>hbase.cluster.distributed</name>
  <value>true</value>
</property>
<property>
  <name>hbase.rootdir</name>
  <value>hdfs://namenode:8020/hbase</value>
</property>
<property>
  <name>hbase.zookeeper.quorum</name>
  <value>zk1,zk2,zk3</value>
</property>

• Startup and verification: Start HDFS first in distributed mode, launch HBase, and open the shell.

start-hbase.sh
hbase shell
status

• Shutdown: Stop HBase cleanly so services flush and close their data structures.

stop-hbase.sh

• Deployment distinction:
  1. Standalone mode: Runs services in one JVM and is appropriate for learning or local development.
  2. Distributed mode: Runs the HMaster and RegionServers as separate processes and is appropriate for production-scale data.

III. Core Database Principles

A. HBase Fundamentals

HBase combines ordered key-value access with automatic horizontal partitioning to handle tables too large for one machine.

• NoSQL character: HBase does not require joins, foreign keys, or a fixed qualifier-level schema; denormalization is therefore common.
• Sorted storage: Rows are ordered by row key, making contiguous range scans efficient but arbitrary non-key searches expensive.
• Horizontal scaling: As a table grows, regions split and can be assigned to additional RegionServers.
• Sparse representation: An absent cell occupies no stored value, so different rows may contain different qualifiers without storing NULL.
• Versioning: A cell may retain multiple values distinguished by timestamps, subject to the column family’s configured version count.
• Atomicity boundary: Mutations to cells within one row can be atomic, enabling consistent updates to related values in that row.
• Appropriate workloads: Billions of rows, high write rates, key-based lookups, and sequential row ranges align with HBase’s design.
• Limitations: HBase is not ideal for small datasets, ad hoc relational analytics, joins, or full-table searches lacking a suitable row-key strategy.

IV. Logical Data Organization

A. HBase Data Model

The HBase data model identifies every stored value through a hierarchical address consisting of table, row key, family, qualifier, and timestamp.

• Table: A named collection of sorted rows, such as students.
• Row key: An arbitrary byte array that uniquely identifies a row; keys such as student001 determine physical sort order.
• Column family: A predefined storage group, such as info or marks; compression, TTL, versions, and storage files are managed at this level.
• Column qualifier: A dynamic field inside a family, written as family:qualifier, such as info:name.
• Cell: The intersection of a row and fully qualified column; its address can be represented as:

(table, row key, column family, qualifier, timestamp) → value

• Timestamp: A 64-bit version identifier, normally assigned from write time in milliseconds unless the client supplies one.
• Versions: Repeated writes to the same cell create timestamped versions, but retention depends on family settings such as VERSIONS.
• Worked example: A value "Asha" stored at row student001, column info:name, and timestamp 1710000000000 is one version of that cell.
• Design consequence: Families should be few and stable because each family has separate storage files; qualifiers may be numerous and dynamic.

V. Distributed Processing Structure

A. HBase Architecture

HBase architecture separates coordination, region management, persistent storage, and client routing so that reads and writes can scale across machines.

• HMaster: Assigns regions, balances cluster load, coordinates schema changes, and handles RegionServer failures; ordinary data traffic does not normally pass through it.
• RegionServer: Hosts multiple regions and serves client reads and writes for their row-key ranges.
• Region: A contiguous range of table rows; growing regions split into daughter regions, permitting redistribution.
• ZooKeeper: Supports coordination, failure detection, and discovery of the hbase:meta region location.
• hbase:meta: A system table containing mappings from user-table row ranges to RegionServers.
• Write path:
  1. A RegionServer appends the mutation to the write-ahead log (WAL).
  2. It updates the in-memory MemStore.
  3. MemStore contents are later flushed as immutable HFiles in HDFS.
• Read path: The RegionServer checks memory and relevant HFiles, using block indexes, caches, and Bloom filters where configured.
• Compaction: Minor and major compactions merge HFiles, reduce file count, and eventually remove eligible deleted or expired cells.
• Failure tolerance: WAL data can be replayed after a RegionServer failure, while HDFS replication protects persistent files.

VI. Shell-Based Table Operations

A. General Commands in Apache HBase

General HBase shell commands support namespace inspection, table definition, data manipulation, scanning, and administration.

• Environment inspection:

status
version
list

• Table creation: Define at least one column family.

create 'students', 'info', 'marks'
describe 'students'

• Insert or update: put writes one cell; writing the same address again creates a newer version.

put 'students', 'student001', 'info:name', 'Asha'
put 'students', 'student001', 'marks:bigdata', '87'

• Point lookup: get retrieves one row and can restrict the result to selected columns.

get 'students', 'student001'
get 'students', 'student001', {COLUMN => 'info:name'}

• Range retrieval: scan iterates through rows in row-key order.

scan 'students'
scan 'students', {STARTROW => 'student001', STOPROW => 'student100'}

• Deletion: delete removes a cell version or column target according to the command form, while deleteall removes the row’s cells.

delete 'students', 'student001', 'marks:bigdata'
deleteall 'students', 'student001'

• Administrative operations: Tables must generally be disabled before being dropped.

disable 'students'
enable 'students'
truncate 'students'
disable 'students'
drop 'students'

• Operational caution: scan, truncate, and destructive commands can be expensive or irreversible on large production tables.

VII. Row-Key Selection

A. Prefix filtering in HBase

Prefix filtering returns rows whose row keys begin with a specified byte sequence, exploiting the ordered nature of the row-key space.

• Shell form: PrefixFilter is commonly attached to a scan.

scan 'events', {FILTER => "PrefixFilter('user42#')"}

• Result behavior: Keys such as user42#001 and user42#2025-03-01 match, while user7#001 does not.
• Java API form:

Filter filter = new PrefixFilter(Bytes.toBytes("user42#"));
Scan scan = new Scan();
scan.setFilter(filter);

• Efficiency principle: Prefixes correspond to contiguous row-key ranges, allowing HBase to avoid returning unrelated rows; explicit start/stop rows may improve range planning further.
• Use case: Composite keys such as customerId#timestamp support scanning all records for one customer.
• Limitation: A suffix or substring search is not equivalent to a prefix scan and may require a redesigned key, index, or external search system.
• Design caution: Monotonically increasing or heavily shared prefixes can concentrate writes in one region and create a hotspot.

VIII. Cell-Value Selection

A. Single value column filtering in HBase

Single-column value filtering keeps or rejects entire rows by comparing a specified column’s value with a comparator.

• Filter class: SingleColumnValueFilter names a family and qualifier, applies an operator, and uses a comparator such as BinaryComparator.
• Shell example:

scan 'students', {
  FILTER => "SingleColumnValueFilter('marks','bigdata',>=,'binary:80')"
}

• Comparison caution: Values are byte arrays; lexical comparison of text numerals can produce unexpected order, so fixed-width encoding or suitable binary numeric encoding is preferable.
• Missing-column behavior: By default, rows missing the tested column may pass the filter; filterIfMissing=true is needed when absence should reject the row.
• Version behavior: The filter commonly checks the latest version first; API options such as setLatestVersionOnly(false) alter version handling.
• Java construction:

SingleColumnValueFilter f = new SingleColumnValueFilter(
    Bytes.toBytes("info"),
    Bytes.toBytes("status"),
    CompareOperator.EQUAL,
    Bytes.toBytes("active"));
f.setFilterIfMissing(true);

• Limitation: This is server-side filtering, not a secondary index; HBase may still scan many rows before discarding nonmatches.

IX. Time-Based Data Retention

A. TTL for columns in HBase

Time to live automatically expires old cells, but the standard schema-level TTL is configured for an entire column family rather than an individual qualifier.

• Unit: Column-family TTL is specified in seconds; for example, 86400 represents one day.
• Creation syntax:

create 'logs', {NAME => 'events', TTL => 86400}

• Alteration syntax:

alter 'logs', {NAME => 'events', TTL => 604800}

• Expiration rule: A cell becomes expired when its timestamp is older than the configured retention interval relative to current time.
• Visibility and removal: Expired cells are normally excluded from reads, but their physical bytes may remain in HFiles until compaction removes them.
• Qualifier granularity: To assign different schema-level TTLs to logical columns, place them in separate column families because TTL is a family property.
• Cell-level option: Supported client mutation APIs can attach a TTL to individual cells, usually in milliseconds; it cannot extend retention beyond the family-level TTL.
• Operational risk: Client-supplied timestamps, clock errors, and very short TTLs can cause data to expire earlier or later than intended.