An Approach for Dynamic Scheduling of Data Analysis Algorithms

Dynamic scheduling of a set of algorithms is a key problem for data analysis platform. In this paper, we propose an approach to efficiently execute and monitor algorithms. Our approach classifies all algorithms into timing tasks, realtime tasks C and equal interval times tasks and configures them separately. An intelligent strategy performs configuration checking for algorithms before scheduling them dynamically. The execution of each algorithm is monitored and controlled according to configuration and feedbacked operating information. Based on this approach, we develop an intelligent data analysis platform with more than 100 algorithms. By stable running for months, our approach is proved to be accurate and effective, and can be applied in many platforms.