Containerized Workflow Builder for Kubernetes

Abstract

Docker and Kubernetes have recently dominated the whole cloud-native technology ecosystem and speeded up the containerization process of workflows. By optimizing the task scheduling order, workflow scheduling algorithms signifi-cantly improve the execution efficiency of workflows. However, existing works do not support workflow scheduling following the optimized task scheduling order on Kubernetes. How to ensure the consistency of workflow scheduling algorithms and Kubernetes scheduler in task scheduling order is challenging. In this paper, we propose a Containerized Workflow Builder (CWB) for Kubernetes, a framework able to implement con-tainerization of workflows in a two-level scheduling scheme and connect workflow scheduling algorithms to Kubernetes, ensuring the consistency of the task scheduling order. It uses the Informer component to watch the underlying resource events of Kubernetes, providing the event trigger mechanism to respond to event callback in real-time and implement containerization of workflows by the Client-go package. Experimental results show that our proposed CWB ensures the consistency of the workflow scheduling algorithms and Kubernetes scheduler in the task scheduling order. Compared with the state-of-the-art, the CWB achieves better performance in terms of average task pod execution time, average workflow lifecycle, and resource usage rate.