Response Time Characterization of Microservice-Based Systems

In pursuit of faster development cycles, companies have favored small decoupled services over monoliths. Following this trend, distributed systems made of microservices have grown in scale and complexity, giving rise to a new set of operational problems. Even though this paradigm simplifies development, deployment, management of individual services, it hinders system observability. In particular, performance monitoring and analysis becomes more challenging, especially for critical production systems that have grown organically, operate continuously, and cannot afford the availability cost of online benchmarking. Additionally, these systems are often very large and expensive, thus being bad candidates for full-scale development replicas. Creating models of services and systems for characterization and formal analysis can alleviate the aforementioned issues. Since performance, namely response time, is the main interest of this work, we focused on bottleneck detection and optimal resource scheduling. We propose a method for modeling production services as queuing systems from request traces. Additionally, we provide analytical tools for response time characterization and optimal resource allocation. Our results show that a simple queuing system with a single queue and multiple homogeneous servers has a small parameter space that can be estimated in production. The resulting model can be used to accurately predict response time distribution and the necessary number of instances to maintain a desired service level, under a given load.