Towards Efficient Resource Allocation for Distributed Workflows Under Demand Uncertainties

Scheduling of complex scientific workflows on geographically distributed resources is a challenging problem. Selection and scheduling of a subset of available resources to meet a given demand in a cost efficient manner is the first step of this complex process. In this paper, we develop a method to compute cost-efficient selection and scheduling of resources under demand uncertainties. Building on the techniques of Sample Average Approximation and Genetic Algorithms, we demonstrate that our method can lead up to \(24\%\) improvement in costs when demand uncertainties are explicitly considered. We present the results from our preliminary work in the context of a high energy physics application, the Belle II experiments, and believe that the work will equally benefit other scientific workflows executed on distributed resources with demand uncertainties. The proposed method can also be extended to include uncertainties related to resource availability and network performance.