Time-Series Data Analytics Using Spark and Machine Learning

This work presents a scalable architecture capable to provide real-time analysis over large-scale time-series data. Spark streaming, Spark MLlib and machine learning methods are combined to process and analyse the data streams. A high performance training model is automatically built and applied for the time-series forecasting. In order to validate the proposed architecture, authors developed a prototype system to predict the average energy consumption at real-time (estimated from 6 K Irish home- and business consumers) from 30 to 90 min ahead. The results show the best prediction was done with a convolutional neural network model, where the Mean Absolute Error and Root Mean Square Error were 7.5% and 10.5% correspondingly.