8000 GitHub - BRIKIAchraf/cdc-lakehouse-nyc: Local CDC lakehouse pipeline for NYC TLC taxi data with PostgreSQL, Debezium, Kafka and Python. · GitHub
[go: up one dir, main page]

Skip to content

About

Local CDC lakehouse pipeline for NYC TLC taxi data with PostgreSQL, Debezium, Kafka and Python.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

13 Commits

Folders and files

Repository files navigation

About

Data engineering portfolio project designing a replayable CDC lakehouse for NYC taxi analytics, from PostgreSQL and Debezium through Kafka, Spark, Hudi, Snowflake and dbt.

CDC Lakehouse NYC

Portfolio architecture for a change-data-capture and analytics platform built around New York City taxi data.

Objective

Create a replayable data platform that can ingest historical NYC TLC files, capture source changes and publish trusted analytical datasets.

Target architecture

NYC TLC Parquet → Python replay loader → PostgreSQL
                                      ↓
                              Debezium CDC
                                      ↓
                                    Kafka
                                      ↓
                         Spark Structured Streaming
                                      ↓
                              Apache Hudi on S3
                                      ↓
                            Snowflake → dbt marts

Open-Meteo weather data is planned as an enrichment source through Airbyte. Data quality, lineage and monitoring are treated as first-class concerns.

Data layers

Layer Grain Purpose
Bronze One CDC event Append-only source payload and replay
Silver One current trip Canonical schema, deduplication and upsert
Gold Trip or hourly aggregate Certified measures for analytics

Engineering principles

  • replayable ingestion and controlled reprocessing;
  • idempotent upserts using business and technical keys;
  • versioned event contracts and backward-compatible evolution;
  • checks for uniqueness, timestamps, geographic bounds and business rules;
  • Kafka lag, freshness, rejected records and processing-time metrics;
  • secrets outside Git and least-privilege access.

Planned repository layout

src/ingestion/       Historical replay and source loading
src/streaming/       Spark CDC processing and Hudi upserts
src/enrichment/      Weather enrichment
dbt/                 Staging, intermediate and mart models
infra/               Docker, Terraform and service configuration
docs/                Architecture and operating decisions

Status

Blueprint in active implementation. The architecture and data contracts are documented; execution jobs, integration tests and deployment manifests are being added incrementally.

Roadmap

  • Add local PostgreSQL, Kafka, Debezium and object storage stack.
  • Implement NYC TLC replay with schema validation.
  • Add CDC contracts and consumer tests.
  • Implement Spark streaming and Hudi upserts.
  • Add dbt models, data-quality checks and observability.
  • Add CI, secret scanning and cloud deployment documentation.

Author

Achraf BRIKI · LinkedIn · Portfolio

Delivery profile

Status: Blueprint
Demo: Architecture walkthrough and planned local CDC stack; no public hosted demo yet.
Prerequisites: Docker, PostgreSQL, Kafka, Debezium, Spark and S3-compatible storage.
Validation: Replay, CDC ordering, duplicate handling, schema evolution, Hudi upserts and dbt tests.
Performance plan: Measure rows/second, CDC lag, batch latency, Hudi throughput, dbt runtime and storage growth.
Roadmap: Runnable infrastructure, sample fixtures, Spark/Hudi jobs, dbt checks, CI and benchmarks.

About

Local CDC lakehouse pipeline for NYC TLC taxi data with PostgreSQL, Debezium, Kafka and Python.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages

0