Eunomia: Efficiently Eliminating Abnormal Results in Distributed Stream Join Systems

With the emergence of big data applications, stream join systems are widely used in extracting valuable information among multi-source streams. However, providing completeness of processing results in a large-scale distributed stream join system is challenging because it is hard to guarantee the consistency among all instances. We show through experiments that the abnormal result can make the quality of achieved data unacceptable in practice.In this paper, we propose Eunomia, a novel distributed stream join system which leverages an ordered propagation model for efficiently eliminating abnormal results. We design a light-weighted self-adaptive strategy to adjust the structure in the model according to the dynamic stream input rates and workloads. It can improve the scalability and performance significantly. We implement Eunomia and conduct comprehensive experiments to evaluate its performance. The results show that Eunomia eliminates abnormal results to guarantee the completeness, improves the system throughput by 25% and reduces the processing latency by 74% compared to state-of-the-art designs.