Green Gates — Data Engineering Portfolio

The thesis

The boring, critical work that production data teams live on.

No dashboards on a toy dataset. Three systems — orchestrated batch ELT, real-time streaming, and data-quality monitoring — each seeding its own deterministic data and running end to end from an empty state. A project is declared done only when its automated gates are verified, not when the code merely looks right.

Words & code — Anish Guntreddi · Postgres-centered local stack

Read the systems The code

warehouselab — make run ✓ PASS

Fig. 1 — one command, empty to verified marts.

The Systems

Features 01–03

Three modes of data engineering, three modes of failure tamed. Each feature lists its architecture, its stack, and the ledger of verification gates that gate “done.”

01Batch ELT

WarehouseLab

Raw events to analytics-ready marts, via an orchestrated and tested ELT pipeline.

raw→ dbt staging→ dbt marts→ Metabase ·Dagster DAG

A deterministic generator lands raw events in Postgres. dbt cleans and types them into staging, then computes business marts — daily active users, revenue by day, a user dimension. Dagster runs the DAG on a daily schedule; the whole thing materializes from empty with make run.

StackPostgreSQL 16 · dbt · Dagster · Metabase · Docker

Source ↗ README ↗

Verification ledger4/4 verified

make run goes empty → populated marts

dbt build — 70/70 tests pass

golden value: DAU = 57, revenue = 1175.29

dbt docs — lineage builds clean

02Real-time

StreamPulse

Process live events and compute real-time windowed metrics, with alerting.

producer→ Redpanda→ processor→ metrics→ alerter

A producer streams synthetic events into a Redpanda topic. A processor maintains tumbling windows — events per minute, conversion rate per five-minute window — and writes them to a metrics store. A live dashboard renders them; an alerter fires on a threshold breach. The windowing logic is pure and unit-tested to exact aggregates.

StackRedpanda · stream processor · PostgreSQL · Streamlit

Source ↗ README ↗

Verification ledger4/4 verified

unit tests assert exact aggregates on a batch

integration run populates metrics in-window

alert fires on a forced breach (rows = 4)

docker compose up brings the pipeline up

03Data quality

DataGuard

Check datasets for schema, null, range, freshness & drift — then score and alert.

expectations→ check engine→ scores→ FastAPI→ Streamlit

A pluggable engine runs checks — schema, null, range, freshness, and distribution drift — against Postgres tables. Drift uses PSI (primary) and KS (secondary); results roll into a deterministic per-table quality score. A FastAPI service serves them; a Streamlit dashboard shows scores, failing checks, and trends. Every check is a pure, tested function.

StackPSI/KS drift · FastAPI · Streamlit · PostgreSQL

Source ↗ README ↗

Verification ledger4/4 verified

each check covered with pass + fail fixtures

drift flags shifted (PSI 4.99), passes stable

quality score deterministic for fixed inputs

dashboard renders against seeded results

The gates are the definition of done — verified, not vibes.

— On building data systems that converge

Specifications

The shared spine

A Postgres-centered local stack keeps the projects coherent and the setup friction low. Heavy runtimes are containerized, so a clean clone runs identically anywhere.

Storage

PostgreSQL 16

landing + marts
metrics store
quality results

Transform

dbt + SQL

staging models
tested marts
lineage docs

Move

Dagster · Redpanda

scheduled DAGs
Kafka streaming
windowed state

Serve

FastAPI · Streamlit · Metabase

quality API
live dashboards
BI on marts

The Method

Why the loop converges

Tight v1 scope, a verification harness, and review steps that read the gates. Build one project at a time; let the gates declare it done; then move on.

Scope

tight v1, explicit out-of-scope

Generate

deterministic seed data

Build

models · processors · checks

Test

unit + golden values

Run

one command, empty → done

Verify

gates green = ship