WRITING / NOTES FROM THE FORGE

Data Engineering

16 articles tagged Data Engineering.

Partitioning and Clustering: How to Make Warehouse Queries Faster

Data partitioning for faster queries: how partition pruning works, pick partition keys, avoid the small files problem, and when clustering or Z-ordering helps.

Sep 19, 20267 min read

Data Engineering File Formats Big Data

Parquet vs Avro vs ORC vs CSV: Choosing a Data File Format

Parquet vs Avro vs ORC vs CSV compared: row vs columnar, compression, predicate pushdown, schema evolution, and which format fits streaming or analytics.

Sep 17, 20267 min read

Data Engineering Data Quality Data Observability

Data Quality: How to Test and Monitor Your Data Pipelines

Learn data quality testing for pipelines: the six quality dimensions, SQL tests, data contracts, anomaly monitoring, and alerting that catches bad data early.

Sep 15, 20267 min read

Data Engineering Data Pipelines Reliability

Data Pipeline Best Practices: Building Reliable, Idempotent Pipelines

Data pipeline best practices for reliable systems: idempotency, safe backfills, partitioned writes, retries, schema evolution, observability, SLAs, and cost.

Sep 13, 20267 min read

Data Engineering ETL Data Pipelines

ETL vs ELT: Key Differences and When to Use Each

ETL vs ELT explained: where transformation happens, cost, governance, and tooling trade-offs, plus a simple framework for choosing the right data pipeline.

Sep 11, 20267 min read

Data Engineering CDC Streaming

Change Data Capture (CDC) Explained: Syncing Databases in Real Time

Change data capture explained: log-based vs query-based vs trigger-based CDC, how connectors stream changes, ordering, schema changes, and the outbox pattern.

Jul 30, 20266 min read

Data Engineering dbt Analytics Engineering

dbt Explained: Transforming Data in the Warehouse With SQL

What is dbt? Learn how models, ref(), sources, tests, materializations, incremental models, and lineage docs help you transform warehouse data with SQL.

Jul 26, 20266 min read

Data Engineering Airflow Orchestration

Apache Airflow Explained: How to Orchestrate Data Pipelines

An Apache Airflow tutorial for data pipeline orchestration: DAGs, tasks, operators, the scheduler, retries, backfills, and idempotent tasks with TaskFlow.

Jul 22, 20267 min read

Data Engineering Apache Spark Big Data

Apache Spark Explained for Beginners: How Distributed Processing Works

Apache Spark explained for beginners: how the driver, executors, partitions, lazy evaluation, and shuffles work, plus a practical PySpark DataFrame example.

Jul 18, 20267 min read

Data Engineering Data Modeling Data Warehouse

Slowly Changing Dimensions (SCD) Explained: Types 1, 2, and 3

Slowly changing dimensions explained: SCD Type 1, 2, and 3 with a SQL MERGE example, surrogate keys, effective dates, and current flags for accurate history.

Jul 16, 20267 min read

Data Engineering Data Modeling Data Warehouse

Star Schema vs Snowflake Schema: Data Modeling for Analytics

Star schema vs snowflake schema explained with facts, dimensions, grain, SQL examples, and clear guidance on which data model fits your analytics warehouse.

Jul 14, 20266 min read

Data Engineering Streaming Data Pipelines

Batch vs Stream Processing: How to Choose for Your Data Pipeline

Batch vs stream processing explained: latency, windows, event time, late data, and exactly-once caveats, plus how to choose the right model for your pipeline.

Jul 10, 20266 min read