Filling the Gap: A Tool to Automate Parameter Estimation for Software Performance Models Weikun Wang, Juan F. P´ erez, Giuliano Casale Department of Computing Imperial College London weikun.wang11@imperial.ac.uk September 1, 2015 1/13 W. Wang. Filling the Gap: A Tool to Automate Parameter Estimation for Software Performance Models
Motivation ◮ DevOps - a recent trend in Software engineering ◮ Bridges the gap between software development and operations ◮ Use performance models for QoS analysis ◮ Accurate parametrization is challenging 2/13 W. Wang. Filling the Gap: A Tool to Automate Parameter Estimation for Software Performance Models
FG tool ◮ Continuous performance model parametrization ◮ Advanced estimation algorithms ◮ Statistical inference from monitoring data ◮ Application QoS report generation 3/13 W. Wang. Filling the Gap: A Tool to Automate Parameter Estimation for Software Performance Models
FG Components ◮ FG Local DB: monitoring data storage ◮ FG Analyzer: statistical analysis ◮ FG Actuator: performance model update ◮ FG Reporter: application performance report 4/13 W. Wang. Filling the Gap: A Tool to Automate Parameter Estimation for Software Performance Models
FG Architecture Computing FG Cluster FG Report FG Local FG DB Analyzer FG QoS Actuator Model Deployment Monitoring Monitoring Module History DB Platform 5/13 W. Wang. Filling the Gap: A Tool to Automate Parameter Estimation for Software Performance Models
Resource Demand Definition: the cumulative execution time a request seizes from a server, excluding contention ◮ An important parameter of queueing models ◮ Difficult to obtain directly ◮ Extensive monitoring poses overhead 6/13 W. Wang. Filling the Gap: A Tool to Automate Parameter Estimation for Software Performance Models
Supported Demand Estimation Algorithms ◮ Complete Information (CI) ◮ Gibbs sampling with Queue Lengths (GQL) ◮ MINPS/FMLPS ◮ Extended Regression-Based approach (ERPS) ◮ FCFS ◮ Utilization-Based Regression/Optimization (UBR/UBO) 7/13 W. Wang. Filling the Gap: A Tool to Automate Parameter Estimation for Software Performance Models
CI and GQL CI ◮ Uses full trace: ts. of arrivals and departures ◮ Poses additional overhead for intensive monitoring GQL ◮ Requires queue length samples, i.e. number of requests at the server ◮ Estimates demand with Bayes’ theorem ◮ Uses Gibbs sampling to obtain demand 8/13 W. Wang. Filling the Gap: A Tool to Automate Parameter Estimation for Software Performance Models
MINPS/FMLPS MINPS/FMLPS ◮ MINPS: a maximum likelihood (ML) method ◮ FMPLS: ML method with fluid approximation ◮ Both requires response times and queue lengths (arrival) 9/13 W. Wang. Filling the Gap: A Tool to Automate Parameter Estimation for Software Performance Models
ERPS and FCFS ERPS ◮ Requires response time and queue length (arrival) ◮ Linear regression to obtain demand FCFS ◮ Estimation for FCFS servers ◮ Requires response time and queue length (arrival) ◮ Linear regression to obtain demand 10/13 W. Wang. Filling the Gap: A Tool to Automate Parameter Estimation for Software Performance Models
Summary: required monitoring data Data Required Algorithm Full trace CI Utilization UBR Throughput UBO Queue length GQL Response Times MINPS Queue length (arrival) ERPS FCFS 11/13 W. Wang. Filling the Gap: A Tool to Automate Parameter Estimation for Software Performance Models
Comparison between Demand Estimation Algorithms 70 10 3 60 10 2 50 10 1 Execution time (s) 40 Error (%) 10 0 30 10 -1 20 10 -2 10 0 10 -3 CI MINPSERPS GQL UBR UBO FMLPS CI MINPSERPS GQL UBR UBO FMLPS Algorithms Algorithms (a) Error (%) (b) Execution time (s) Most accurate: CI Fastest: UBR 12/13 W. Wang. Filling the Gap: A Tool to Automate Parameter Estimation for Software Performance Models
Discussions ◮ i) The feedback for different demand estimation algorithms: e.g. ◮ How much monitoring information can be timely brought to the developers? ◮ Which metric is the easiest or most readily available? ◮ Which metric poses the least overhead? ◮ ii) How to correlate the resource IDs as well as the request types for inconsistent design time model/deployment model/monitoring data? 13/13 W. Wang. Filling the Gap: A Tool to Automate Parameter Estimation for Software Performance Models
Recommend
More recommend
