Provisioning big data applications as services on containerised cloud: a microservices-based approach

We present the fundamental challenges for dynamic provisioning of big data applications. The findings are based on our previous experience in this domain, as well as a comprehensive study on a selected set of state-of-the-art tools in the big data ecosystem. We then incorporate these findings in a framework aiming at dynamically provisioning big data applications as services on containerised cloud. The innovations behind the framework are to optimise the whole lifecycle of big data applications in a holistic manner by the adoption of microservices(μServices) methodologies. The feasibility of our approach is verified through a case study of provisioning a large-scale user traffic data processing application in a private cloud environment backed by Kubernetes. We also show that while hosting big data applications in containerised cloud can significantly eliminate the presumed complexity of deployment and operation, it in the meantime also comes with a certain amount of cost in terms of learning curve and traceability. Our research helps technical decision makers to assess the adoption of microservices for big data applications more objectively.