Motivation
In the fast paced world of data analytics, most analysts take hours or days to identify meaningful insights that can be used in visualizations.
Process
This project involved a complete overhaul of traditional data analytic workflows and rewrote it using modern tools like Storm and Redis. The goal was to aid the new paradigm of visualization. I made an end to end functioning prototype of this new model.
Challenges
The key challenge was the ability to ingest data at the same rate it was being produced. At the same time the system had to also create a data summary, extract meaning and showcase that meaning in a comprehensive visualization. This was achieved by the use of Storm and Redis.
Value Addition
Variations of this have been implemented in advanced versions of Adobe’s Digital Marketing platform. With this we were able to process and showcase the data within about 30 seconds after the data point had been encountered by the monitoring service. A paper was published regarding this technology.
Links
