Data lake architecture

Overview

A data lake is a central repository that stores raw data of any shape (structured tables, JSON events, logs, images) cheaply in object storage such as Amazon S3, in open file formats like Parquet. Unlike a data warehouse, which requires a schema before loading (schema-on-write), a data lake stores data first and applies structure when it is read (schema-on-read).

Modern lakes are organized in layers, often called bronze (raw), silver (cleaned and joined), and gold (business-ready aggregates). A catalog tracks tables and schemas, and query engines (Spark, Athena, Trino) read the files directly. Open table formats such as Apache Iceberg and Delta Lake add ACID transactions, schema evolution, and time travel, turning the lake into a lakehouse.

When to use it

• Storing large volumes of diverse data for analytics and machine learning.
• Keeping raw history cheaply for reprocessing.
• Decoupling storage from compute so multiple engines share the same data.
• Building analytics on top of event streams and CDC.

Where it shows up in interviews

Where it is used in real software

Key terms

Schema-on-read

Structure applied when querying, not when loading.

Parquet

Columnar file format with compression, ideal for analytics.

Partitioning

Organizing files by keys such as date so queries skip irrelevant data.

Table format (Iceberg, Delta)

Metadata layer adding ACID transactions and schema evolution to files.

Medallion architecture

Bronze (raw), silver (clean), gold (aggregated) layers.

How it works, step by step

1. 1
   Ingest raw data

   Batch loads, CDC from databases, and streams (Kafka, Kinesis) land in the bronze zone.

2. 2
   Catalog it

   Register tables, schemas, and partitions in a catalog (Glue, Hive Metastore, Unity Catalog).

3. 3
   Clean and conform

   Deduplicate, validate, and join into silver tables.

4. 4
   Aggregate for consumers

   Build gold tables for dashboards, reporting, and features.

5. 5
   Query and govern

   SQL engines and ML jobs read the tables; access control, lineage, and retention policies apply.

Data lake vs data warehouse vs lakehouse

Key differences

Lakehouses combine cheap open storage with warehouse-like reliability.

Implementation

1-- Silver table in Iceberg, partitioned by day2CREATE TABLE silver.page_views (3  event_id STRING,4  user_id STRING,5  url STRING,6  country STRING,7  event_time TIMESTAMP8)9PARTITIONED BY (days(event_time))10TBLPROPERTIES ('table_type' = 'ICEBERG');11 12-- Idempotent upsert from raw events (deduplicate by event_id)13MERGE INTO silver.page_views t14USING (SELECT DISTINCT event_id, user_id, url, country, event_time FROM bronze.raw_events WHERE dt = '2026-09-26') s15ON t.event_id = s.event_id16WHEN NOT MATCHED THEN INSERT *;17 18-- Gold aggregate; partition pruning reads only one day of files19INSERT OVERWRITE gold.daily_views20SELECT date(event_time) AS day, country, count(*) AS views, approx_count_distinct(user_id) AS uniques21FROM silver.page_views22WHERE event_time >= TIMESTAMP '2026-09-26' AND event_time < TIMESTAMP '2026-09-27'23GROUP BY 1, 2;

Complexity and performance

Trade-offs

Variants and related techniques

Common mistakes

• Millions of tiny files.

  Fix: Batch writes and run compaction jobs into larger Parquet files (128 MB-1 GB).

• No partitioning or poor partition keys.

  Fix: Partition by commonly filtered columns (date), avoiding high-cardinality keys.

• Personal data copied everywhere.

  Fix: Classify, mask, and apply retention and access policies from bronze onward.

Interview questions

Practice problems