Catching Incidents Before Customers Do: Anomaly Detection at Telecom Scale
At telecom scale, the difference between a quiet night and a bad one is often a signal nobody saw in time. Working at Orange Morocco, I built a real-time pipeline that processed more than 4 million records a day and turned that firehose into early warnings, catching 25% more incidents before they reached customers.
The problem: scale plus urgency
Telecom data is huge, fast, and noisy. Buried inside it are the early signs of problems that become customer complaints hours later. The goal was to surface those signs while there was still time to act, without drowning the operations team in false alarms.
Moving the data
The backbone was a set of modular PySpark jobs orchestrated by Apache Airflow. Rather than one monolithic script, each job did a clear piece of work — extract, clean, aggregate — so the pipeline stayed debuggable as it grew. A unified data lake replaced a patchwork of fragmented sources, and curated SQL layers exposed clean, query-ready metrics to Power BI through REST APIs.
Layered anomaly detection
No single model catches every kind of anomaly, so I combined several:
- Isolation Forest for point anomalies — records that are individually strange.
- Autoencoders to learn a compact model of normal and flag reconstructions that miss.
- Time-series methods for anomalies that only reveal themselves against recent history.
Layering them meant the system caught different failure modes while keeping false positives low enough that the alerts stayed trustworthy — and an alert people trust is the only kind worth sending.
The results, and why they held
KPI latency dropped from hours to minutes, and the pipeline caught 25% more incidents before customer impact. Just as important, an MLOps setup kept the models fresh as traffic patterns shifted, because an anomaly detector that is never retrained slowly stops detecting anomalies. The lesson I carry from it: at scale, reliability and retraining discipline matter as much as the model itself.