Most operational data goes stale before anyone can act on it. This pipeline keeps it live: an end-to-end streaming ETL system that ingests data, processes it in real time, and flags anomalies the moment they appear.
Records are pulled from a source database by a dedicated extractor, streamed through Apache Kafka, transformed and cleaned with PySpark, and scored by a scikit-learn anomaly-detection model before being loaded into a destination store and pushed to Power BI for live dashboards. Each stage is an isolated, replaceable component.
The whole system is orchestrated by Apache Airflow, monitored with Prometheus, and packaged with Docker Compose so it starts and scales as one unit – production-minded data engineering, not a notebook demo.