Concurrency vs parallelism - System Design Guide

Overview

Concurrency is about dealing with many things at once; parallelism is about doing many things at once. A concurrent program structures work as independent tasks that can make progress in overlapping time periods, even on one CPU core by switching between them. A parallel program executes tasks at the same instant on multiple cores.

The distinction guides design. I/O-bound systems (web servers waiting on databases) benefit from concurrency, handling thousands of waiting requests with few threads. CPU-bound work (video encoding, ML training) benefits from parallelism across cores or machines. Many systems need both.

A chef vs a kitchen crew

One chef juggling three dishes (chopping while water boils, stirring while the oven heats) is concurrency. Three chefs each cooking a dish at the same time is parallelism.

When to use it

Choosing async I/O vs thread pools vs multiprocessing.
Explaining why Node.js handles many connections on one thread.
Sizing thread pools for I/O-bound vs CPU-bound work.
Interview discussions of scalability on a single machine.

Where it shows up in interviews

I/O-bound scaling

Recognize it when: requests spend most time waiting on network or disk.

Design a high-concurrency API gateway
Design a web crawler

CPU-bound scaling

Recognize it when: heavy computation per task.

Design a video transcoding service
Design a batch analytics job

Where it is used in real software

Node.js

A single-threaded event loop serves thousands of concurrent connections because most time is spent waiting on I/O.

Go goroutines

Go multiplexes many goroutines onto OS threads, giving concurrency by default and parallelism across cores with GOMAXPROCS.

MapReduce and Spark

Split data across many machines and process partitions in parallel.

Key terms

Concurrency: Structuring work as overlapping tasks.
Parallelism: Executing tasks simultaneously on multiple processors.
I/O-bound: Limited by waiting on network or disk.
CPU-bound: Limited by computation.
Amdahl's law: Speedup is limited by the portion that must run sequentially.

How it works, step by step

1
Classify the workload
Measure whether time goes to waiting (I/O) or computing (CPU).
2
I/O-bound - add concurrency
Async I/O, event loops, or large pools of lightweight threads.
3
CPU-bound - add parallelism
Threads or processes up to the number of cores, or distribute across machines.
4
Split work into independent pieces
Minimize shared state and coordination.
5
Measure speedup
Watch for contention, lock waits, and the sequential portion.

Three tasks on one core vs three cores

Task A

Task B

Task C

Core 1

STEP 1Concurrency on one core: A runs until it waits for I/O.

Picking the right approach

Four workloads

Workload	Bottleneck	Approach
API calling 3 services per request	I/O waiting	Concurrency (async or virtual threads)
Image resizing	CPU	Parallelism (process or thread pool per core)
Web crawler	Network I/O	High concurrency with rate limits
Training a model	CPU/GPU	Data and model parallelism across devices

NOWWorkload: API calling 3 services per request | Bottleneck: I/O waiting | Approach: Concurrency (async or virtual threads)

Adding threads to CPU-bound work beyond core count does not help; adding concurrency to I/O-bound work helps a lot.

Implementation

1// Concurrency: overlap three I/O calls on one thread2async function productPage(id: string) {3  const [product, reviews, stock] = await Promise.all([4    fetch(`/api/products/${id}`).then((r) => r.json()),5    fetch(`/api/reviews/${id}`).then((r) => r.json()),6    fetch(`/api/stock/${id}`).then((r) => r.json()),7  ]);8  return { product, reviews, stock }; // total time ~= slowest call, not the sum9}

Complexity and performance

Ideal parallel speedupup to N on N cores

Limited by the sequential fraction (Amdahl).

CPU-bound pool size~number of cores

More threads add context switches.

Trade-offs

Complexity

Concurrent and parallel code introduces races, deadlocks, and nondeterminism; prefer message passing and immutable data.

Overhead

Splitting tiny tasks across cores can be slower than running them sequentially due to scheduling and coordination costs.

Variants and related techniques

Async/await

Language-level concurrency for I/O without blocking threads.

Data parallelism

Same operation on partitions of data (SIMD, GPUs, Spark).

Common mistakes

Using many threads for CPU-bound Python.
Fix: The GIL serializes Python bytecode; use multiprocessing or native libraries.
Blocking calls inside an event loop.
Fix: One blocking call stalls every request; use async APIs or offload to workers.

Interview questions

What is the difference between concurrency and parallelism?

Concurrency is structuring a program so multiple tasks make progress in overlapping time, even on one core by interleaving. Parallelism is executing multiple tasks at the same instant on multiple cores. Concurrency is about structure; parallelism is about execution.

Why can Node.js handle many connections with one thread?

Most request time is spent waiting on I/O. The event loop starts I/O operations and handles their completions as callbacks, so one thread can interleave thousands of waiting requests; it is concurrent but not parallel for JavaScript code.

Practice problems

Problem	Difficulty	What it trains
Parallelize independent API calls	Easy	Promise.all and timeouts.
Size pools for a mixed CPU and I/O service	Medium	Workload classification.