Leveraging Endpoint Flexibility When Scheduling Coflows Across Geo-distributed Datacenters

Authors:
Wenxin Li Dalian University of Technology, P.R. China
Xu Yuan University of Louisiana at Lafayette, USA
Keqiu Li Tianjin University, P.R. China
Heng Qi Dalian University of Technology, P.R. China
Xiaobo Zhou Tianjin University, P.R. China

Abstract:

Coflow scheduling is crucial to improve the communication performance of data-parallel jobs, especially when these jobs running in the inter-datacenter networks with limited and heterogeneous link bandwidth. However, prior solutions on coflow scheduling assume the endpoints of flows in a coflow to be fixed, making them insufficient to optimize the coflow completion time (CCT). In this paper, we focus on the problem of jointly considering endpoint placement and coflow scheduling to minimize the average CCT of coflows across geo-distributed datacenters. We first develop the mathematical model and formulate a mixed integer linear programming (MILP) problem to characterize the intertwined relationship between endpoint placement and coflow scheduling, and reveal their impact on the average CCT. Then, we present SmartCoflow, a coflow-aware optimization framework, to solve the MILP problem without any prior knowledge of coflow arrivals. In SmartCoflow, we first apply an approximate algorithm to obtain the endpoint placement and scheduling decisions for a single coflow. Based on the single-coflow solution, we then develop an efficient online algorithm to handle the dynamically arrived coflows. To validate the efficiency and practical feasibility of SmartCoflow, we implement it as a real-world coflow scheduler based on the Varys open-source framework. Through experimental results from both a small-scale testbed implementation and large-scale simulations, we demonstrate that SmartCoflow can achieve significant improvement on the average CCT, when compared to the state-of-the-art scheduling-only method.

You may want to know: