Unit 1: Introduction to Spark - Subjective Questions

MapReduce limitations:

• Repeated disk I/O: MapReduce writes intermediate results to disk after each map and reduce stage, which increases latency.
• Poor performance for iterative algorithms: Machine learning, graph processing, and scientific computations repeatedly use the same data. MapReduce reloads the data during every iteration.
• High latency: MapReduce is designed mainly for batch processing, so it is unsuitable for applications that require near-real-time results.
• Complex programming model: Developers must divide applications into map and reduce functions, even when the problem does not naturally fit this structure.
• Limited interactive analysis: Interactive queries are slow because each query may initiate a separate job.
• Inefficient data sharing: Sharing data between multiple processing stages is difficult and expensive because intermediate data is materialized on disk.

Apache Spark addresses many of these issues by supporting in-memory processing, directed acyclic graphs, interactive queries, and multiple workloads within one framework.

Batch analytics processes a large, stored dataset at scheduled intervals. Real-time analytics processes data continuously as it arrives.

Batch analytics is suitable when immediate results are not necessary. Real-time analytics is essential when a delayed response may reduce business value or create risk. Spark supports both approaches through batch processing and stream processing APIs.

Stream processing analyzes data continuously as it is generated instead of waiting for data to be stored in large batches.

Important applications include:

• Fraud detection: Financial transactions can be analyzed immediately to identify suspicious behavior.
• Internet of Things: Sensor data can be monitored to detect equipment failures or abnormal readings.
• Social media analysis: Posts and user interactions can be processed to identify trends and sentiment.
• Log and event monitoring: System logs can be examined for failures, security threats, and performance problems.
• Online recommendations: User activity can update product or content recommendations immediately.
• Telecommunications: Network traffic can be analyzed to detect congestion and service problems.
• Real-time dashboards: Business metrics can be displayed as events occur.

The main benefits are low latency, continuous insight, rapid response, and the ability to make decisions using the most recent data.

In-memory processing stores frequently used data in the main memory of a cluster instead of repeatedly reading it from disk.

It improves performance in the following ways:

• Lower access latency: Memory access is substantially faster than disk access.
• Reduced disk operations: Intermediate results can remain in memory instead of being written and read repeatedly.
• Efficient iteration: Algorithms that reuse the same data, such as machine learning and graph algorithms, can access cached data quickly.
• Faster interactive queries: Users can run multiple queries on cached datasets without reconstructing the data each time.
• Improved data sharing: Several processing stages can reuse the same in-memory dataset.

Spark uses abstractions such as Resilient Distributed Datasets and DataFrames to cache data across cluster nodes. If memory is insufficient, Spark can spill data to disk, preserving functionality with reduced performance.

Apache Spark is a distributed data-processing framework based on a master-worker architecture.

• The driver program contains the application logic and coordinates execution.
• The cluster manager allocates resources to the application. It may be Spark Standalone, YARN, Kubernetes, or another supported manager.
• Worker nodes provide computational resources.
• Executors run tasks on worker nodes and store data in memory or on disk.
• Spark converts application operations into jobs, stages, and tasks using a directed acyclic graph.

Spark transformations are generally lazy, meaning they are not executed immediately. They are recorded in a logical plan. When an action such as count, collect, or save is called, Spark builds an execution plan and schedules tasks across the cluster.

This architecture enables parallel processing, fault recovery, data caching, and support for different workloads.

Important features of Apache Spark include:

• In-memory computation: Data can be cached in memory to accelerate repeated operations.
• Distributed processing: Workloads are divided into tasks and executed across multiple machines.
• Fault tolerance: Lost partitions can be recomputed using lineage information.
• Lazy evaluation: Transformations are optimized and executed only when an action requires a result.
• Unified analytics: Spark supports batch processing, SQL, streaming, machine learning, and graph processing.
• Multiple language interfaces: Applications can be developed using Scala, Java, Python, and R.
• Scalability: Spark can run on a single machine or across large clusters.
• DAG execution engine: Complex workflows are optimized as directed acyclic graphs rather than being restricted to only map and reduce phases.
• Interactive processing: Users can submit queries and commands through shells and notebooks.
• Broad storage support: Spark can access HDFS, local files, cloud storage, NoSQL databases, and other data sources.

Benefits of Apache Spark:

• High speed: In-memory caching and optimized execution reduce the time required for many workloads.
• Productivity: High-level APIs simplify distributed application development.
• Unified platform: Batch jobs, SQL queries, streaming applications, machine learning, and graph workloads can be developed using one framework.
• Cost effectiveness: Spark can run on commodity hardware and does not require a separate engine for every workload.
• Flexibility: It supports multiple programming languages, deployment modes, and storage systems.
• Fault tolerance: Spark can recover lost data partitions through lineage and recomputation.
• Scalability: Resources can be increased as data volume and processing requirements grow.
• Advanced optimization: DataFrame and SQL workloads benefit from query optimization and efficient execution.
• Interactive analysis: Analysts can explore data more quickly than with traditional disk-heavy batch systems.

These benefits make Spark suitable for both exploratory analysis and production data pipelines.

Typical installation steps:

1. Install Java: Install a compatible Java Development Kit and verify it using java -version.
2. Download Spark: Obtain a pre-built Spark distribution from the official Apache Spark website.
3. Extract the archive: Unpack the downloaded file into a suitable directory.
4. Configure environment variables: Set SPARK_HOME to the Spark directory and add its bin directory to the system PATH.
5. Verify the installation: Run spark-shell for Scala or pyspark for Python.
6. Test Spark: Create a small collection and perform an action such as counting its elements.
7. Run a sample application: Submit an application using spark-submit.

For standalone local use, Spark can run in local mode with a command such as local[*], where the asterisk allows Spark to use the available processor cores. A production cluster installation requires additional configuration for worker nodes and a cluster manager.

Local mode is convenient for experimenting with Spark applications. Standalone cluster mode uses Spark's own cluster manager, in which a master allocates executors and tasks to worker nodes. The programming model is similar in both modes, but the available resources and operational complexity differ.

Spark is often faster and more flexible, while Hadoop MapReduce remains useful for reliable, large-scale batch processing with disk-based execution.

Iterative machine-learning algorithms repeatedly process the same training data. Examples include clustering, classification, and gradient-based optimization.

In MapReduce, each iteration normally performs the following operations:

• Read the training data from storage.
• Execute map and reduce tasks.
• Write intermediate results to disk.
• Start the next iteration by reading the data again.

This repeated disk access causes high execution time. Spark improves the process by allowing the training dataset to be cached in memory across iterations. Each iteration can then reuse the cached partitions. Spark's DAG scheduler also optimizes the sequence of transformations and reduces unnecessary synchronization between stages.

Spark additionally provides machine-learning libraries with distributed algorithms. As a result, iterative workloads generally achieve lower latency and better resource utilization, provided that sufficient memory is available.

Lazy evaluation means that Spark does not immediately execute transformations such as map, filter, or select. Instead, it records them and waits until an action requires a result.

Its effects include:

• DAG construction: Spark builds a directed acyclic graph representing the complete computation.
• Pipeline optimization: Compatible transformations can be combined into a single stage.
• Reduced data movement: Spark can avoid unnecessary shuffles and transfers between nodes.
• Limited computation: Only the data and operations required for the requested result are executed.
• Better fault recovery: The recorded lineage shows how lost partitions can be recomputed.

An action such as count, collect, or write triggers execution. Lazy evaluation therefore separates the description of a computation from its actual execution and gives Spark more information for optimization.

A Resilient Distributed Dataset, or RDD, is an immutable, partitioned collection of elements distributed across the nodes of a cluster.

Its important properties are:

• Distributed: Data is divided into partitions that can be processed in parallel.
• Immutable: Operations create new RDDs rather than modifying existing ones.
• Lazy: Transformations are recorded and executed only when an action is called.
• Resilient: Spark stores lineage information describing how each partition was derived.
• Cacheable: Frequently used RDDs can be stored in memory or on disk.

If a partition is lost because of a node failure, Spark uses lineage to recompute only that partition from its original data and transformations. This avoids requiring every intermediate result to be replicated. RDDs are particularly useful when applications need fine-grained control over distributed data and fault recovery.

Spark provides a common execution platform for different data-processing workloads.

• Batch processing: Spark reads bounded datasets from sources such as files, HDFS, databases, or object storage. It processes the complete dataset and produces a final result.
• Stream processing: Spark Structured Streaming reads unbounded data from sources such as Kafka, files, or network systems. It processes data incrementally as new records arrive.
• Common APIs: DataFrames, SQL expressions, transformations, and output operations can be used in both batch and streaming applications.
• Shared libraries: The same platform can combine streaming data with machine learning, SQL, and other analytics.
• Fault handling: Checkpointing, recovery mechanisms, and structured execution help maintain reliable streaming jobs.

This unified approach reduces the need to maintain separate systems and allows organizations to reuse data models, application logic, and operational practices.

The Hadoop ecosystem contains several specialized components:

• HDFS: Distributed storage for large files.
• YARN: Cluster resource management and scheduling.
• MapReduce: Disk-oriented batch processing engine.
• Hive: SQL-based data warehousing and querying.
• Pig: Data-flow scripting for batch analysis.
• HBase: Distributed column-oriented database.
• Sqoop and Flume: Data import and collection tools.

Spark provides a processing engine and libraries rather than replacing every Hadoop component:

• Spark Core: Distributed execution and resource coordination.
• Spark SQL: Structured data processing and SQL queries.
• Structured Streaming: Stream processing.
• MLlib: Machine learning.
• GraphX: Graph computation.

Spark can use HDFS for storage and YARN for resource management. Therefore, Spark and Hadoop are often complementary. Hadoop supplies durable storage and cluster services, while Spark supplies a faster and more flexible computation engine.

Spark provides specialized libraries for different analytical requirements:

• Spark SQL: Processes structured and semi-structured data using SQL, DataFrames, and Datasets. It supports schema-based operations and query optimization.
• Structured Streaming: Processes continuously arriving data using a structured, DataFrame-oriented programming model. It supports operations such as filtering, aggregation, joins, and windowing.
• MLlib: Provides distributed machine-learning algorithms, feature-processing tools, pipelines, and evaluation utilities.
• GraphX: Supports graph-parallel computation for relationships among entities, such as social networks and web links.

All these components run on Spark Core. This integration makes it possible to use one application for activities such as reading data, transforming it with SQL, applying a machine-learning model, and analyzing relationships in a graph.

The company should use a hybrid Spark architecture:

• Use Structured Streaming to read transactions continuously from a source such as Kafka.
• Apply validation, filtering, enrichment, and fraud-detection rules as events arrive.
• Store alerts and short-term aggregates in a low-latency data store or dashboard system.
• Persist the incoming transactions in durable storage such as HDFS or cloud object storage.
• Use a scheduled Spark batch job to read the accumulated data at the end of each month.
• Perform joins, aggregations, and historical comparisons to create the monthly sales report.

This approach matches the processing model to each requirement. Immediate transaction analysis requires low latency, while monthly reporting benefits from complete historical data and high-throughput batch processing. Spark allows both workloads to share APIs, infrastructure, and data-processing logic.

Transformations create a new dataset from an existing dataset but are evaluated lazily. Examples include:

• map
• filter
• flatMap
• select
• groupBy

Actions produce a result or write data and trigger execution. Examples include:

• count
• collect
• first
• reduce
• show
• write

For example, a Spark program may first filter records using a transformation and then call count as an action. Spark records the filter operation, but actual processing starts only when count is invoked. This separation allows Spark to optimize the complete computation before running tasks on the cluster.

Partitioning divides a dataset into smaller parts so that multiple tasks can process the data in parallel. A partition is generally handled by one task.

Shuffling is the redistribution of data across partitions, usually required by operations such as groupByKey, joins, and some aggregations. Records with related keys must be brought together, possibly across different worker nodes.

Excessive shuffling reduces performance because it:

• Transfers data over the network.
• Creates additional disk and memory operations.
• Increases serialization and deserialization overhead.
• Can produce uneven partitions and data skew.
• Introduces stage boundaries and task coordination costs.

Performance can be improved by choosing suitable partitioning, using combiners or reduce-side aggregation, filtering data before a shuffle, and using broadcast joins when one dataset is sufficiently small.

The speedup obtained with Spark is workload-dependent rather than automatic. It can be understood using the following factors:

• Let represent time spent reading and writing intermediate data to disk.
• Let represent data-transfer time.
• Let represent actual computation time.
• Let represent memory access and caching overhead.

For a disk-oriented job, execution time can be approximated as:

For a Spark job that successfully caches reused data, the approximate time may be:

The speedup is therefore:

The speedup is greatest when the workload is iterative, intermediate data is reused, and sufficient memory is available. It may be modest for a single-pass batch job dominated by computation or input/output. Spark can also lose performance because of garbage collection, memory pressure, serialization, or excessive shuffling.