Unit 1: Introduction to Spark

I. Foundations of Apache Spark

Apache Spark is an open-source, distributed computing engine designed for large-scale data processing. It originated at the University of California, Berkeley’s AMPLab in 2009 and became an Apache top-level project in 2014. Spark distributes data and computation across a cluster while supporting batch processing, streaming, SQL, machine learning, and graph analysis through a unified platform.

A. Defining Characteristics

Spark follows a cluster-computing model in which a driver coordinates parallel work performed by executors.

• Distributed execution: A dataset is divided into partitions that can be processed concurrently on different CPU cores or cluster nodes.
• Driver program: The driver creates the SparkSession, constructs execution plans, schedules jobs, and collects results or status information.
• Executors: Executor processes run tasks and store cached data on worker nodes for the lifetime of an application.
• Cluster manager: Spark obtains computing resources through its standalone manager, Hadoop YARN, or Kubernetes.
• Lazy evaluation: Transformations such as filter() and select() build a logical plan; computation begins only when an action such as count() or write() is invoked.
• Fault tolerance: Lost partitions can be reconstructed from lineage, which records the transformations used to derive them.
• Unified APIs: Spark provides APIs for Scala, Python, Java, R, and SQL, with DataFrames serving as the principal structured-data abstraction.

II. Hadoop MapReduce

A. Limitations of MapReduce in Hadoop

Hadoop MapReduce is reliable for large batch jobs, but its disk-oriented, rigid execution model is inefficient for iterative and low-latency workloads.

• Repeated disk I/O: Intermediate output is commonly materialized to local storage, while output between jobs is written to HDFS.
  • A pipeline of three MapReduce jobs may repeatedly read and write the same dataset.
  • Disk and network operations are much slower than reusing data already held in memory.
• High job latency: Starting mappers, reducers, containers, and shuffle operations introduces substantial overhead, making MapReduce unsuitable for responses expected in seconds or milliseconds.
• Rigid two-stage model: Computation must be expressed mainly through map and reduce phases, even when an algorithm naturally requires joins, iteration, filtering, or multiple processing stages.
• Inefficient iterative processing: Machine-learning algorithms repeatedly process the same data. In MapReduce, each iteration may reload that data from HDFS.
  • For 20 iterations over a 100 GB dataset, the input may be scanned approximately 20 times.
• Complex pipelines: Multi-stage analysis requires several chained jobs, temporary directories, serialization steps, and explicit dependency management.
• Limited interactive analysis: Analysts cannot efficiently issue repeated exploratory queries because every query begins another high-overhead batch job.
• No native continuous processing: Classic MapReduce processes bounded files rather than continuously arriving event streams.
• Programming burden: Even operations such as word counting require mapper, reducer, key-value, configuration, and job-submission code.

A simplified MapReduce word-count flow is:

map(line):
    for each word in line:
        emit(word, 1)

reduce(word, counts):
    emit(word, sum(counts))

Here, word is the grouping key and counts is the collection of emitted values associated with that key.

III. Data-Processing Models

A. Comparison of batch vs. real-time analytics

Batch analytics processes a bounded collection after it has accumulated, whereas real-time analytics processes events soon after they arrive.

1. Batch analytics:

   • Input: Operates on finite datasets such as one day of sales records stored in HDFS or object storage.
   • Latency: Results may be produced after minutes or hours; throughput and completeness are usually more important than immediate response.
   • Scheduling: Jobs commonly run hourly, nightly, or at the end of a reporting period.
   • Applications: Payroll calculation, monthly billing, historical trend analysis, and data-warehouse loading fit the batch model.
   • Processing model: A job reads a defined input snapshot, transforms it, writes results, and terminates.

2. Real-time analytics:

   • Input: Operates on unbounded events, such as sensor readings, payment transactions, or server logs.
   • Latency: Results are expected in milliseconds or seconds so that the system can react while an event remains relevant.
   • Continuous operation: Processing remains active and consumes new records as they become available.
   • Applications: Fraud alerts, operational monitoring, recommendation updates, and equipment-failure detection require rapid results.
   • Processing model: Records are processed individually or in small micro-batches, often using event-time windows.

• Explicit contrast: A nightly sales report can wait for all stores to upload data, but card-fraud detection must evaluate a transaction before or immediately after authorization.
• Trade-off: Batch systems simplify completeness and recomputation; real-time systems must additionally manage late events, ordering, checkpoints, state, and recovery.

IV. Modern Processing Techniques

A. Application of stream processing and in-memory processing

Stream processing handles continuously generated events, while in-memory processing accelerates repeated access to active datasets and intermediate results.

1. Stream processing:

   • Event ingestion: Events may enter through Apache Kafka, cloud message services, files, or network sources.
   • Transformations: A stream can be filtered, grouped, aggregated, enriched, or joined with reference data.
   • Windowing: Infinite input is divided into meaningful intervals, such as five-minute transaction windows.
   • Stateful processing: The system preserves values across events, such as the running number of failed logins per account.
   • Structured Streaming: Spark represents a stream as an incrementally updated table and executes DataFrame or SQL operations as new data arrives.
   • Concrete application: A monitoring pipeline can count HTTP 500 responses in each five-minute window and send excessive counts to an alert table.

2. In-memory processing:

   • Caching: Frequently reused DataFrames can be retained with cache() or persist() rather than reread from storage.
   • Iterative algorithms: Clustering and classification algorithms benefit because each iteration can reuse cached training data.
   • Interactive queries: Analysts can issue several queries against a cached dataset with less storage I/O.
   • Fallback behavior: In-memory does not mean memory-only; Spark can spill partitions to disk when configured storage capacity is insufficient.
   • Serialized execution: Spark’s optimizer and execution engine reduce unnecessary data movement and generate efficient physical plans.

events = spark.readStream.format("kafka").option(
    "subscribe", "transactions"
).load()

events.writeStream.format("console").start()

In this example, events is an unbounded streaming DataFrame and start() begins continuous query execution.

V. Apache Spark Platform

A. Features and benefits of Spark

Spark combines multiple analytics workloads in one engine and offers abstractions that are more expressive than the classic MapReduce interface.

• Processing speed: Memory reuse, optimized execution plans, pipelined transformations, and reduced intermediate disk writes can significantly improve suitable workloads.
• DataFrames and SQL: Structured data can be queried through SQL or language APIs, while Spark’s optimizer selects operations such as scans, joins, and aggregations.
• Resilient Distributed Datasets: RDDs provide immutable, partitioned collections with lineage-based recovery and low-level transformation control.
• Unified libraries:
  • Spark SQL: Structured queries and integration with formats such as Parquet, JSON, CSV, Hive tables, and JDBC sources.
  • Structured Streaming: Incremental processing using DataFrame and SQL semantics.
  • MLlib: Distributed feature processing and machine-learning algorithms.
  • GraphX: Graph-parallel computation through Scala APIs.
• Ease of development: Python, Scala, Java, R, and SQL APIs allow users to choose a suitable language while using the same execution engine.
• Lazy optimization: Spark examines a chain of transformations before execution and can reduce unnecessary work through predicate pushdown and column pruning.
• Scalability: The same application can run locally for development and on a multi-node cluster with configuration changes.
• Fault recovery: Lineage reconstructs lost RDD partitions, while streaming checkpoints and write-ahead mechanisms support recovery of stateful workloads.
• Storage integration: Spark reads from HDFS, cloud object stores, Hive, Kafka, relational databases, and many connector-supported systems.
• Operational visibility: The Spark web UI displays jobs, stages, tasks, executors, SQL plans, storage use, and shuffle metrics.

VI. Standalone Spark Setup

A. Installation of Spark as a standalone user

A standalone installation requires a supported Java runtime, a Spark binary distribution, environment variables, and either local mode or Spark’s standalone cluster manager.

• Prerequisites: Install a Java version supported by the selected Spark release and verify it with java -version.
• Distribution: Download an official pre-built Spark archive, extract it into a user-owned directory, and avoid requiring administrator privileges.
• Environment: Define SPARK_HOME and add Spark’s bin directory to PATH.

export SPARK_HOME="$HOME/spark"
export PATH="$SPARK_HOME/bin:$PATH"
spark-shell

• Local verification: spark-shell starts a Scala shell, while pyspark starts a Python shell. Local mode can be requested with --master local[*], where * uses all available logical CPU cores.

spark-submit --master "local[*]" application.py

• Standalone master: Start Spark’s built-in master service and note the printed master URL, commonly in the form spark://hostname:7077.

"$SPARK_HOME/sbin/start-master.sh"
"$SPARK_HOME/sbin/start-worker.sh" spark://hostname:7077

• Application submission: Submit an application using the standalone master URL with spark-submit --master spark://hostname:7077 application.py.
• Web interfaces: The master UI commonly uses port 8080, and an active application’s Spark UI commonly uses port 4040.
• Shutdown: Stop services with stop-worker.sh and stop-master.sh.
• Configuration: Memory, cores, logging, and defaults can be adjusted through files under $SPARK_HOME/conf or through spark-submit options.

VII. Ecosystem Perspective

A. Comparison of Spark vs. Hadoop ecosystem

Spark is primarily a processing engine, whereas Hadoop is a broader ecosystem containing storage, resource-management, and batch-processing components.

• Scope: Spark supplies computation libraries and APIs; Hadoop includes HDFS for storage, YARN for resource management, and MapReduce for processing.
• Storage: Spark has no mandatory distributed storage layer. It can process data stored in HDFS, cloud object stores, databases, or local files.
• Execution: Hadoop MapReduce materializes more intermediate data to disk; Spark can pipeline transformations and cache reusable data.
• Workload support: MapReduce emphasizes batch processing, while Spark supports batch, interactive SQL, machine learning, graph processing, and streaming.
• Cluster management: Spark can run on its standalone manager, YARN, or Kubernetes. It can therefore operate inside a Hadoop deployment without replacing HDFS or YARN.
• Latency: Spark is generally better suited to iterative and lower-latency workloads; MapReduce remains effective for long-running, disk-based batch jobs.
• Fault tolerance: HDFS protects stored data through block replication, while Spark recovers computed partitions through lineage and application-level mechanisms.
• Resource use: Spark’s caching can demand substantial RAM, whereas MapReduce can process large workloads with stronger dependence on disk.
• Relationship: Spark and Hadoop are not strict substitutes. A common architecture stores files in HDFS, allocates resources through YARN, and executes analysis through Spark.
• Selection criterion: Spark is preferred for unified and iterative analytics; MapReduce may remain appropriate for stable, simple batch pipelines where latency is unimportant and memory is constrained.