Real-Time User Data Stream Processing And Validation For US Based Auto Assistance Company

Situation

• US based auto assistance company with more than 50 million members
• Client had sensor devices installed in 1000’s of vehicles. These devices transmit information back for tracking overall health of the car and for trigger for assistance
• A lot of data was being generated real time which could potentially be used for providing better services and as input for insurance and planning for future service offerings

Challenges

• A device installed in the vehicle generates a data file with rows for every second of the journey
• Two data files for every journey (from multiple devices) were generated – (i) Stats of the journey from start of the engine to switching off (ii) Exception data wherever vehicle has crossed any pre-set threshold value
• Initially devices were sending partial, incorrect or corrupt data back for downstream processes including user based insurance. Client installed new devices and partnered with Axtria to process real time data, validate and send correct data for subsequent processes

Approach

• Axtria used Apache storm for multiple processing engines installed on multiple machines for parallel processing and validation of data files
• Data validation introduced to clean the data and take forward only relevant data for further analysis and reporting. More than 150 test cases were built to validate real time data in less than a second
• Available information was bucketed into different groups for better analytics and user based insurance

Result

• Improved and faster data management process of real time data
• Data quality for downstream processes improved
• Advanced analytics done on new buckets of information for improving service offerings

VALIDATED DATA BUCKETED INTO PACKETS OF INFORMATION FOR FURTHER ANALYTICS

• Acceleration & deceleration, speed (maximum, minimum and average) and break usage data was used to create a pattern of driving skills of the user
• Latitude/Longitude information over a period of several weeks was plotted to identify most frequent paths taken by the user and areas where maximum need of service is required