Unit 5: Introduction to Apache HBase - Subjective Questions

Apache HBase is an open-source, distributed, column-family-oriented NoSQL database built on top of the Hadoop Distributed File System (HDFS). It is modeled after Google's Bigtable and is designed to store very large, sparse datasets.

Major characteristics:

• Distributed storage: Data is divided across multiple machines.
• Column-family model: Related columns are grouped into column families.
• Horizontal scalability: Capacity can be increased by adding more nodes.
• Strong consistency: Reads return the latest successfully written value.
• Sparse data support: Empty columns consume little or no storage.
• Versioning: Multiple versions of a cell can be maintained using timestamps.
• Real-time access: It supports low-latency random reads and writes over large datasets.

Prerequisites:

• A supported Java Development Kit must be installed.
• JAVA_HOME must point to the Java installation directory.
• The Apache HBase binary distribution must be downloaded and extracted.

Installation steps:

1. Download a compatible HBase release from the Apache HBase website.
2. Extract the archive and move into the HBase directory.
3. Edit conf/hbase-env.sh and configure JAVA_HOME.
4. Edit conf/hbase-site.xml and set hbase.rootdir to a local directory.
5. Keep hbase.cluster.distributed set to false for standalone mode.
6. Start HBase using bin/start-hbase.sh.
7. Open the shell using bin/hbase shell.
8. Verify the installation by running the status command.
9. Stop HBase using bin/stop-hbase.sh.

Standalone mode runs the HBase services and local storage components in a single JVM and is mainly suitable for learning and development.

In distributed mode, HBase stores its persistent data in HDFS and runs its services across multiple machines.

Configuration process:

• Install and configure a Hadoop cluster with working HDFS.
• Install the same compatible HBase version on all HBase nodes.
• Configure JAVA_HOME in hbase-env.sh.
• Set hbase.cluster.distributed to true in hbase-site.xml.
• Set hbase.rootdir to an HDFS path such as hdfs://namenode:8020/hbase.
• Configure hbase.zookeeper.quorum with the hostnames of ZooKeeper nodes.
• List RegionServer hosts in the conf/regionservers file.
• Synchronize configuration files across all nodes.
• Start HDFS, ZooKeeper when externally managed, and then HBase.
• Verify the cluster using the HBase shell status command and the HBase Master web interface.

A distributed setup provides fault tolerance, scalability, load distribution, and persistent storage. Compatibility among Java, Hadoop, HBase, and ZooKeeper versions must be checked before deployment.

The HBase data model organizes information as a distributed, multidimensional sorted map.

• Table: A logical collection of rows, such as students.
• Row key: A unique byte sequence that identifies a row. Rows are stored in lexicographic order by row key.
• Column family: A group of related columns declared when the table is created, such as personal or marks.
• Column qualifier: A column name created within a family, such as personal:name. Qualifiers can be added dynamically.
• Timestamp: Identifies a version of a cell. HBase can assign it automatically or accept a client-supplied value.
• Cell: The value identified by the combination of row key, column family, column qualifier, and timestamp.

A cell can therefore be represented conceptually as:

(row key, column family, column qualifier, timestamp) -> value

For example, student01, personal:name, and a timestamp may identify the cell containing Asha.

Column family:

• It is a physical and logical grouping of related columns.
• It must be declared when the table is created or later added through a schema alteration.
• Storage properties such as compression, TTL, Bloom filters, and maximum versions are generally configured at this level.
• The number of column families should usually remain small.
• Example: info in info:name.

Column qualifier:

• It identifies an individual column within a column family.
• It does not need to be declared in advance.
• Different rows may contain different qualifiers.
• A table can have a very large number of qualifiers.
• Example: name in info:name.

Thus, a column is addressed as family:qualifier. The family determines the storage grouping, while the qualifier identifies a specific attribute inside that group.

The row key determines the physical ordering, region placement, access pattern, and distribution of HBase rows. Since rows are sorted lexicographically, a poor design can create hotspots or require expensive full-table scans.

Recommended practices:

• Design keys around the most common query patterns.
• Keep keys compact because each cell is associated with its row key internally.
• Avoid monotonically increasing prefixes, such as raw timestamps, when they direct all new writes to one RegionServer.
• Use salting, hashing, bucketing, or reversed identifiers when write distribution is required.
• Place frequently searched components toward the beginning of a composite key.
• Use fixed-width or unambiguous separators for composite key components.
• Avoid embedding fields that frequently change.

For example, region#customerId#reverseTimestamp can support regional prefix scans and recent-first ordering. However, salting can improve write distribution while making range scans more complex, so the choice must follow workload requirements.

Major HBase architecture components:

• HMaster: Coordinates administrative operations such as table creation, region assignment, load balancing, and recovery from RegionServer failures.
• RegionServer: Serves read and write requests for the regions assigned to it. It manages MemStores, WAL operations, and HFiles.
• Region: A contiguous range of rows belonging to a table. Regions split as they grow.
• ZooKeeper: Supports coordination, server discovery, Master election, and failure detection.
• HDFS: Provides distributed and fault-tolerant storage for HFiles and write-ahead logs.
• WAL: Records data modifications before they are applied to in-memory structures, enabling recovery after failure.
• MemStore: Holds recent updates in memory for a particular column family before they are flushed.
• HFile: An immutable, sorted file in HDFS that stores HBase data.
• BlockCache: Caches frequently accessed HFile blocks to improve read performance.

Clients locate regions through HBase metadata and then communicate directly with the responsible RegionServer for data operations.

The HBase write path maintains durability while supporting fast writes.

Write sequence:

1. The client determines which RegionServer owns the row's region.
2. The client sends a Put request to that RegionServer.
3. The RegionServer validates the request and appends the modification to the Write-Ahead Log (WAL).
4. After the WAL append is durable, the value is placed in the relevant column family's MemStore.
5. The RegionServer acknowledges the successful write to the client.
6. When the MemStore reaches a threshold, its sorted contents are flushed to an immutable HFile in HDFS.
7. Over time, minor and major compactions combine HFiles and remove eligible obsolete or deleted cells.

The WAL provides recovery if a RegionServer fails before its MemStore is flushed. The MemStore permits low-latency writes because every update does not immediately require creation of an HFile.

During a read, the client first identifies the RegionServer responsible for the requested row and sends it a Get or Scan request.

Read path:

1. The RegionServer checks the MemStore for recent data that has not yet been flushed.
2. It checks the BlockCache for previously read HFile blocks.
3. If required data is not cached, the server reads relevant blocks from HFiles stored in HDFS.
4. Results from these sources are merged according to timestamps, versions, deletions, and filters.
5. The newest visible matching value or requested versions are returned to the client.

Roles:

• MemStore: Supplies recent in-memory updates.
• BlockCache: Reduces repeated disk and HDFS reads.
• HFiles: Store persistent, sorted data.
• Bloom filters and indexes: May help avoid reading HFiles that cannot contain the requested row or column.

Efficient row keys, caching, and a controlled number of HFiles significantly affect read performance.

A region is a horizontal partition of an HBase table containing a contiguous interval of rows. Because table rows are sorted by row key, each region has a start key and an end key.

Region splitting:

• A new table may initially contain one region unless pre-splitting is used.
• As the region grows beyond a configured size, it is divided into two daughter regions.
• The daughter regions cover separate adjacent row-key ranges.
• Splitting enables a growing table to be distributed across additional RegionServers.

Region assignment:

• The HMaster assigns each region to one RegionServer at a time.
• RegionServers directly handle reads and writes for assigned regions.
• The HMaster reassigns regions when a server fails or when balancing cluster load.
• Region metadata is used by clients to locate the correct RegionServer.

Pre-splitting may improve initial write distribution when expected row-key ranges and workload patterns are known.

Apache HBase:

• Uses a distributed column-family data model.
• Supports flexible and sparse columns.
• Scales horizontally across commodity servers.
• Is optimized for massive datasets and row-key-based access.
• Does not provide general-purpose joins or a full SQL interface by itself.
• Provides atomicity primarily at the row level.
• Usually denormalizes data according to access patterns.

Relational database:

• Uses tables with predefined columns and data types.
• Commonly supports SQL, joins, constraints, and relational normalization.
• Traditionally scales vertically, although distributed relational systems also exist.
• Is suitable for complex queries and multi-table transactional workloads.
• Commonly provides broader ACID transaction support.

HBase is appropriate for large-scale, sparse data requiring fast keyed reads and writes. A relational database is generally preferable when joins, referential integrity, ad hoc SQL queries, and multi-row transactions are central requirements.

Consider a table named students with column families info and marks.

• Create the table: create 'students', 'info', 'marks'
• List available tables: list
• Describe the table schema: describe 'students'
• Check whether the table exists: exists 'students'

Explanation:

• create defines the table and its column families.
• list displays tables available in the current HBase namespace.
• describe shows schema information and column-family properties such as versions and TTL.
• exists returns whether the named table is present.

Column families must be specified at table-creation time, whereas column qualifiers such as info:name can be added dynamically when values are inserted.

Assume that the table students already exists.

• Insert or update a cell: put 'students', 's001', 'info:name', 'Ravi'
• Read a row: get 'students', 's001'
• Read a specific column: get 'students', 's001', 'info:name'
• Scan the table: scan 'students'
• Scan selected columns: scan 'students', {COLUMNS => ['info:name']}
• Delete a cell: delete 'students', 's001', 'info:name'

put writes a new timestamped version of a cell. get performs a point lookup using a row key. scan reads a range of rows and should be constrained for large tables whenever possible. delete creates a deletion marker for the specified cell or version; the underlying data is physically removed later when compaction makes it eligible.

For a table named students, common administrative commands include:

• Disable the table: disable 'students'
• Enable the table: enable 'students'
• Add a column family: alter 'students', NAME => 'attendance'
• Change a property: alter 'students', NAME => 'info', VERSIONS => 3
• Truncate the table: truncate 'students'
• Drop the table: drop 'students'

A table must be disabled before it can be dropped because disabling closes its regions and prevents clients from reading or modifying it during the destructive schema operation. Depending on the HBase version and alteration, some schema changes may also require or trigger region reopening.

truncate removes all table data while recreating the table with its schema. drop removes the disabled table definition and its data. These operations must be used carefully because they are destructive.

A prefix filter selects rows whose row keys begin with a specified byte sequence. It is useful when related information is encoded at the start of a row key.

HBase shell command:

scan 'employees', {FILTER => "PrefixFilter('EMP2024')"}

This scan returns rows with keys such as EMP2024001 and EMP2024HR05, while excluding keys that do not start with EMP2024.

Key points:

• The filter operates on row keys, not cell values.
• It can reduce the number of rows returned to the client.
• Prefix-based access is efficient when row-key design keeps related rows together.
• A bounded start and stop row may provide additional scan optimization when exact lexical boundaries can be calculated.
• A poor row-key design may cause hotspotting even if it makes prefix scans convenient.

Both mechanisms are used to retrieve rows whose row keys start with a given prefix.

Using PrefixFilter:

• It is expressed through the general filter syntax.
• Example: scan 'orders', {FILTER => "PrefixFilter('CUST01')"}
• It can be combined with other filters using constructs such as FilterList.

Using ROWPREFIXFILTER:

• It is a convenient scan option for specifying a row-key prefix.
• Example: scan 'orders', {ROWPREFIXFILTER => 'CUST01'}
• It is concise when prefix selection is the only row-key condition.

Comparison:

• Both express prefix-based row selection.
• PrefixFilter is more natural in complex filter expressions.
• ROWPREFIXFILTER is simpler for a basic prefix scan.
• Exact behavior and accepted shell syntax should be confirmed for the installed HBase version.

In either case, the row-key schema must place the searchable component at the beginning of the key.

SingleColumnValueFilter retains rows based on the value of one specified column.

Its main logical parameters are:

• Column family: The family containing the target column.
• Column qualifier: The target column within that family.
• Comparison operator: Examples include =, !=, <, <=, >, and >=.
• Comparator: Determines how the stored bytes are compared, such as a binary comparator.
• Filter-if-missing setting: Determines whether a row lacking the target column should be excluded.
• Latest-version-only setting: Determines whether only the latest version or multiple versions are checked.

Example:

scan 'employees', {FILTER => "SingleColumnValueFilter('info', 'department', =, 'binary:Sales')"}

This selects rows in which info:department matches Sales. By default, rows missing the specified column may still be included, depending on filter configuration, so applications requiring strict matching should explicitly enable filtering of missing columns.

A value filter can be applied as follows:

scan 'employees', {FILTER => "SingleColumnValueFilter('job', 'salary', >, 'binary:50000')"}

Explanation:

• job is the column family.
• salary is the column qualifier.
• > is the comparison operator.
• binary:50000 is the comparator expression.

Important limitation:
HBase stores values as byte arrays. A binary comparator compares bytes lexicographically rather than automatically interpreting them as integers. Therefore, variable-length decimal strings may not follow numeric order. For example, the lexical ordering of 9000 and 50000 differs from their numeric ordering.

To obtain reliable ordering, the application can:

• Encode numeric values using a consistent binary numeric representation and an appropriate comparator, or
• Store fixed-width, zero-padded positive decimal strings, such as 000050000.

Server-side filters reduce returned data, but they are not equivalent to relational indexes and may still require scanning many rows.

Time To Live (TTL) specifies how long cell versions in a column family remain eligible to be read. TTL is usually expressed in seconds and is based on each cell's timestamp.

Configuration examples:

• Create a table with TTL: create 'events', {NAME => 'data', TTL => 86400}
• Alter an existing family: alter 'events', NAME => 'data', TTL => 86400

Here, 86400 seconds represents one day because:

Enforcement:

• Once a cell exceeds the configured TTL, it is considered expired and is normally omitted from reads.
• Expired data may remain physically present in HFiles temporarily.
• Compaction eventually removes expired cells from storage.
• TTL is generally configured per column family, so different families can have different retention periods.
• Cell timestamps must be trustworthy because TTL expiration is calculated from them.

TTL, version retention, deletion, and compaction jointly determine which HBase cells are visible and when their storage is reclaimed.

• TTL: Makes a cell version expired after the configured duration has elapsed from its timestamp.
• Versions: A column family may retain multiple timestamped versions, subject to settings such as VERSIONS and minimum-version policies.
• Delete markers: A delete operation records a tombstone that hides matching cells or versions rather than immediately rewriting HFiles.
• Compaction: Merges immutable HFiles and physically discards data that is eligible for removal, including expired versions and obsolete cells hidden by tombstones.

A cell can become invisible before its bytes are physically removed. Therefore, a drop in visible row count does not imply immediate release of HDFS space. Major compaction can accelerate cleanup but is resource-intensive and should be scheduled carefully. Administrators must coordinate TTL and version settings so that business retention requirements are met without keeping unnecessary historical data.