Reinforcement Learning based Autoscaling for Kafka-centric Microservices in Kubernetes

Abstract

Microservices and Kafka have become a perfect match for enabling the Event-driven Architecture and this encourages microservices integration with various opensource platforms in the world of Cloud Native applications. Kubernetes is an opensource container orchestration platform, that can enable high availability, and scalability for Kafkacentric microservices. Kubernetes supports diverse autoscaling mechanisms like Horizontal Pod Autoscaler (HPA), Vertical Pod Autoscaler (VPA) and Cluster Autoscaler (CA). Among others, HPA automatically scales the number of pods based on the default Resource Metrics, which includes CPU and memory usage. With Prometheus integration, custom metrics for an application can be monitored. In a Kafkacentric microservices, processing time and speed depends on the number of messages published. There is a need for auto scaling policy which can be based on the number of messages processed. This paper proposes a new autoscaling policy, which scales Kafka-centric microservices deployed in an eventdriven deployment architecture, using a Reinforcement Learning model.