Apache Spark Explained for Beginners: How Distributed Processing Works

Apache Spark is a distributed data processing engine that splits a large dataset into partitions and processes those partitions in parallel across a cluster of machines. With Apache Spark explained in one sentence: you describe what you want done to the data, Spark builds an optimized plan, and a driver program coordinates many executors that each work on a slice of the data. Once you understand the driver, executors, partitions, lazy evaluation, and shuffles, most Spark behavior stops being mysterious.

What is Apache Spark?

Spark is an open-source engine for large-scale data processing. It runs batch jobs, SQL queries, streaming workloads (Structured Streaming), and machine learning pipelines using one programming model. You can write Spark code in Python (PySpark), Scala, Java, R, or plain SQL, and the same job can run on your laptop or on a cluster managed by Kubernetes, YARN, or Spark's standalone manager.

The problem Spark solves is simple to state: some datasets are too large, or some computations too slow, for a single machine. Spark lets you treat a dataset spread across hundreds of files and many machines as if it were one table, while it handles distribution, scheduling, and recovery from failures.

How does Apache Spark work?

Every Spark application has the same basic architecture.

The driver

The driver is the process that runs your main program. It creates the SparkSession, turns your code into a logical plan, optimizes it, breaks it into stages and tasks, and schedules those tasks on executors. The driver also collects results that you explicitly bring back, which is why calling collect() on a huge DataFrame can crash it with an out-of-memory error.

Executors

Executors are worker processes launched on cluster nodes. Each executor has a number of cores and a chunk of memory. A core runs one task at a time, and a task processes one partition. If you have, for example, 10 executors with 4 cores each, Spark can run 40 tasks concurrently.

Partitions

A partition is a chunk of your data that one task handles. When Spark reads files, it splits them into partitions based on file sizes and configuration. Partitioning is the single most important factor in Spark performance: too few partitions and cores sit idle; too many tiny partitions and scheduling overhead dominates. Uneven partitions, called skew, mean one task runs far longer than the rest while the whole stage waits for it.

Transformations vs actions in Spark

Spark operations fall into two groups, and the difference explains lazy evaluation.

• Transformations describe a new dataset derived from an existing one: select, filter, withColumn, join, groupBy().agg(). They return a new DataFrame immediately without touching any data.
• Actions ask for a result: count, show, collect, take, and writes such as write.parquet(...). An action forces Spark to actually execute the plan.

What is lazy evaluation?

Because transformations do nothing on their own, Spark can see your whole chain of operations before running anything. The Catalyst optimizer uses that view to push filters closer to the data source, prune columns you never use, reorder operations, and pick join strategies. Reading only the columns you need from a Parquet file, and skipping row groups that cannot match a filter, often saves more time than any amount of cluster tuning.

Lazy evaluation also has a practical consequence: an error in your transformation logic may not appear until an action runs, possibly many lines later. When debugging, trigger small actions like limit(10).show() to surface problems early.

Narrow vs wide transformations

A narrow transformation, such as filter or withColumn, computes each output partition from exactly one input partition. Spark can chain many of them into a single stage. A wide transformation, such as groupBy, distinct, or most joins, needs rows with the same key to end up in the same partition. That requires a shuffle.

What is a shuffle in Spark?

A shuffle redistributes data across the cluster by key. Each task writes its output into buckets, one per target partition, and tasks in the next stage fetch the buckets they need over the network. Shuffles involve disk writes, serialization, and network transfer, so they are usually the most expensive part of a Spark job.

Common ways to reduce shuffle cost:

1. Filter and select columns before joins and aggregations so less data moves.
2. Broadcast small tables in joins, so each executor gets a full copy and the large table never shuffles.
3. Avoid unnecessary repartition calls; use coalesce when you only need fewer partitions.
4. Enable Adaptive Query Execution (on by default in recent Spark 3.x releases), which can merge small shuffle partitions and split skewed ones at runtime.
5. Pick a join or grouping key with reasonably even distribution to limit skew.

Spark DataFrames vs RDDs

The RDD (resilient distributed dataset) is Spark's original low-level abstraction: a distributed collection of arbitrary objects. DataFrames add a schema, named columns, and access to the Catalyst optimizer and the Tungsten execution engine. For almost all data engineering work, DataFrames or Spark SQL are the right choice. They are faster because Spark understands the structure of the data, and they are easier to read. RDDs remain useful for rare cases where you need fine-grained control over unstructured records.

A PySpark example: daily revenue by country

The following job reads order data from Parquet, computes daily revenue per country, and writes the result partitioned by date. Paths are placeholders.

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = SparkSession.builder.appName("daily-revenue").getOrCreate()

orders = spark.read.parquet("s3a://example-bucket/raw/orders/")
countries = spark.read.parquet("s3a://example-bucket/ref/countries/")

daily_revenue = (
    orders
    .filter(F.col("status") == "completed")                   # narrow
    .withColumn("order_date", F.to_date("created_at"))        # narrow
    .join(F.broadcast(countries), on="country_code")          # small table, no shuffle of orders
    .groupBy("order_date", "country_name")                    # wide: triggers a shuffle
    .agg(
        F.sum("amount").alias("revenue"),
        F.countDistinct("customer_id").alias("customers"),
    )
)

daily_revenue.explain()  # print the physical plan without running the job

(
    daily_revenue.write
    .mode("overwrite")
    .partitionBy("order_date")
    .parquet("s3a://example-bucket/marts/daily_revenue/")
)

spark.stop()

Nothing is computed until the write call, which is the action. Calling explain() shows the plan Spark chose, including the BroadcastHashJoin and the Exchange operator that marks the shuffle. The Spark UI then shows the job split into stages at that shuffle boundary, which is the first place to look when a job is slow.

How does Spark handle failures?

Spark tracks the lineage of each DataFrame: the chain of transformations that produced it from source data. If an executor dies, Spark recomputes only the lost partitions by replaying that lineage on another executor, rather than restarting the whole job. Shuffle output written to disk also helps later stages recover without redoing earlier ones. This is why Spark jobs should read from durable, replayable sources such as files in object storage or a Kafka topic.

When to use Apache Spark

Spark is a good fit when data is too large for one machine, when you need to join and aggregate many large files, or when you want one engine for batch and streaming on a data lake. It is often overkill for a few gigabytes that a single-node tool or a warehouse SQL query could handle faster and with less operational overhead. For a broader view of where Spark fits, see data lake architecture and batch vs stream processing.

Key takeaways

• The driver plans and schedules; executors run tasks; each task processes one partition.
• Transformations are lazy and build a plan; actions trigger execution.
• Wide transformations cause shuffles, which are usually the most expensive part of a job.
• Prefer DataFrames and Spark SQL over RDDs so the optimizer can do its work.
• Filter early, select only needed columns, and broadcast small tables to cut shuffle cost.
• Use explain() and the Spark UI to understand what your job actually does.

Frequently asked questions

Is Apache Spark a database?

No. Spark is a processing engine, not a storage system. It reads data from sources like object storage, HDFS, databases, or Kafka, processes it in memory and on local disk, and writes results back out. Table formats such as Delta Lake, Apache Iceberg, and Apache Hudi add table semantics on top of files that Spark can read and write.

Is PySpark slower than Scala Spark?

For DataFrame and SQL operations, PySpark and Scala generally run the same optimized plan on the JVM, so performance is usually similar. The gap appears with plain Python UDFs, which move rows between the JVM and Python workers. Prefer built-in functions, and use vectorized pandas UDFs when custom Python logic is unavoidable.

How many partitions should a Spark job have?

There is no universal number. A common starting point is a few tasks per available core, with partition sizes large enough to avoid scheduling overhead but small enough to fit comfortably in executor memory. Adaptive Query Execution can adjust shuffle partition counts at runtime, which removes much of the manual tuning.

What is the difference between cache and persist?

cache() stores a DataFrame using the default storage level, while persist() lets you choose the level, such as memory only or memory and disk. Both are lazy and take effect on the next action. Cache only data you reuse several times, and call unpersist() when you are done to free executor memory.