Selected data engineering projects: the ones with an ingest point, a processing stage, and something serving results at the end.
Spatial Selection Bias in Broadband Data
MSc thesis
Ookla + RURA data→
Observation model→
Cross-fit calibration→
Corrected estimates
Compares Ookla's self-selected speed tests against RURA's full subscriber census, then models the probability a location enters the crowdsourced sample to correct for it. Thirty quarters of measurements, terrain, and technology mix pulled from 28 regulator PDFs feed an out-of-fold calibration step.
Real-Time Fraud Detection
Source
Kafka→
Spark Structured Streaming→
Random Forest→
Postgres + HDFS
An end-to-end pipeline that simulates live transaction traffic and scores it in-flight. Predictions split two ways: Postgres for a low-latency Streamlit dashboard, HDFS as a data lake for batch analytics and retraining.
eCommerce Purchase Prediction
Source
HDFS→
Kafka (WebHDFS)→
Spark Streaming (MLlib)→
MySQL→
Django
A fully containerized clickstream pipeline that predicts whether a shopping session will end in a purchase while the session is still in flight, surfaced on a live dashboard. Built with a five-person team for AUCA's Big Data Essentials course.
Smart Energy Grid Analytics
Source
Sensors (MQTT / EMQX)→
TimescaleDB→
Grafana + Streamlit
Ingestion and analytics pipeline for smart-grid sensor data. Devices publish over MQTT through EMQX into a Postgres/TimescaleDB store, surfaced through Grafana boards and an interactive Streamlit dashboard.
Hadoop vs. Spark, Head-to-Head
Source
Hadoop MapReduce
vs
PySpark
A Master's-level benchmark comparing MapReduce against Spark on the IEEE-CIS fraud dataset (590,540 transactions, 394 columns), measuring performance and implementation complexity, not just output.