Skip to content

Data Engineering Patterns

Pick a pattern. Pick a stack. Pull the branch. Start building.

What this is

Nine opinionated, production-informed reference implementations of common data engineering patterns — each available in 2–3 idiomatic stack variants. Every variant is a self-contained boilerplate starter: real code, a local dev stack, and a README that explains when to use it and when not to.

Start here

  • Landscape

    Where each pattern fits in the modern data engineering landscape. Read this first if you are choosing a pattern.

    Landscape

  • Decision Matrix

    Two tables: pick a pattern by use-case characteristics, then pick a stack variant by trade-off criteria.

    Decision Matrix

Patterns

  • Batch Lakehouse

    Scheduled Spark jobs writing to open table formats. Highest throughput, simplest operations, no sub-minute latency.

    Batch Lakehouse

  • Streaming Lakehouse

    Continuous Flink or Spark Streaming pipelines writing to Iceberg or Delta. Sub-minute latency with lakehouse query semantics.

    Streaming Lakehouse

  • ELT / Warehouse

    dbt transforms data already loaded into a warehouse. Right when your warehouse is the system of record.

    ELT / Warehouse

  • CDC Pipeline

    Debezium captures database writes at the WAL level and streams them into a processing engine. Zero polling, sub-second capture.

    CDC Pipeline

  • Lambda Architecture

    Parallel streaming (low latency) and batch (high accuracy) paths merged at query time. Use only when both are genuinely required.

    Lambda Architecture

  • Federated Query

    Trino or Presto queries across multiple storage systems without data movement. Right for polyglot data estates.

    Federated Query

  • ML Feature Pipeline

    Compute and serve ML features from a feature store. Separates feature engineering from model training and serving.

    ML Feature Pipeline

  • Graph Processing

    Spark GraphX or Neo4j+Spark for graph-native algorithms. Right when relationships are the query, not the filter.

    Graph Processing

  • Workflow Orchestration

    Airflow, Prefect, or Dagster for scheduling and coordinating data pipelines. The glue between every other pattern.

    Workflow Orchestration

  • :material-rdf:{ .lg .middle } Semantic Medallion

    Extends Batch Lakehouse, CDC Pipeline, and Federated Query with RDF triples and SPARQL for entity deduplication and semantic reasoning across polyglot sources.

    Semantic Medallion

Architecture decision records

Key design decisions for the patterns library:

Browse all ADRs.