PerformoBot, please help me! Chatbot-supported Performance - PowerPoint PPT Presentation

performobot@gmail.com @PerformoBot “PerformoBot, please help me!” Chatbot-supported Performance Evaluation Samuel Beck @samu_beck Lasse Merz 06.11.2019 Christoph Zorn Fabian Beck @beck_fabian Leonel Merino @leonel_merino Dušan Okanović @okanovic_d André van Hoorn @andrevanhoorn

Motivation Performance Evaluation System ChatBot Eve Colin Mia 2 Chatbot-supported Performance Evaluation

Goals ● Evaluate the applicability of a chatbot to facilitate performance evaluations. ● Gather impressions from performance experts and novices about using a chatbot in this context. I want to perform a Perform a load test with I want to test Eve’s load test for Eve’s 100 users and measure system. system. response time. 3 Chatbot-supported Performance Evaluation

Approach User Interaction Language Processing Analysis & Reporting Eve Mia Colin PerformoBot 4 Chatbot-supported Performance Evaluation

Demo 5 Chatbot-supported Performance Evaluation

Demo 6 Chatbot-supported Performance Evaluation

: Research Questions Study How can a chatbot help users do a 01 Task performance evaluation through a load test? + How do users interact with 02 PerformoBot? Survey What educational effect does 03 PerformoBot have on participants? What are the differences between 04 inexperienced users and experts in 47 participants the above categories? 7 Chatbot-supported Performance Evaluation

Study: Task + Survey Performance Evaluation Test System chat Participant evaluate 8:00 min PerformoBot Survey 8 Chatbot-supported Performance Evaluation

Study: Results Median of 10 Average of interactions per 3.64 minutes 47 participants participant session time No educational 87.2% finished effect the task Too little guidance Automatic execution 80.9% correctly fallback mechanisms & answered the & report creation scenario missing keywords Complex load testing capabilities Explanations 9 Chatbot-supported Performance Evaluation

Study: Inexperienced Users vs Experts Helpfulness of Sentiment PerformoBot 22 novices and Very much positive neutral negative 25 experts 18.2% 48.0% Novices and 18.2% experts were equally able to 8.0% successfully complete the task 63.7% 44.0% Novices evaluated PerformoBot more positive and Not very much helpful 10 Chatbot-supported Performance Evaluation

Conclusion Mia Colin ● More performance concerns Helpful for novice/inexperienced users ● Improved usability features Eve Needs advanced functionality for experts https://declare-project.github.io/Performo BotSurveyWebsite/index.html 11 Chatbot-supported Performance Evaluation

Load test intents and entities Intent: Load test Newbie Output context: loadtest Entities: ● Analysis type Training Phrases: Please perform a load test ● I want to create a load test. for me. ● Help me to define a load test. Great, let us set up a load test together. I ● I want to perform a load test. need to know what load you want to use, the domain to be tested and the metric which should be measured. Intent: Define a load test Output context: loadtest Record the response time for 100 users on Entities: www.example.com. ● Load ● Domain ● Metric Alright I created a load test for you with ● Analysis type (optional) the following parameters: load: 100 ● Duration (optional) users, domain: www.example.com, metric: response time. If you are Training Phrases: satisfied I can execute the load test. ● Loadtest with 55 users Otherwise, you can change the ● I want one with 3 hours, 22 users parameters, add more parameters from and response time. the parameter list or reset and start ● Create a load test with 30 users at over. www.myservice.de and measure server status. 12 Chatbot-supported Performance Evaluation

Performance Report 1 3 2 13 4 Chatbot-supported Performance Evaluation

References Icons from flaticon (https://www.flaticon.com/) made by dave-gandy (question mark), monkik (people), freepik, eucalyp, smashicons (analysis, computers, clocks), flat-icons (performobot), roundicons (check mark, x, smileys) chatbot gif: https://cdn.dribbble.com/users/37530/screenshots/2937858/drib_blink_bot.gif Messages gif: https://i.gifer.com/origin/98/98447b873b927d46f752e9e0fc9c2910_w200.gif Okanović et al., <under review>: Dušan Okanović, Samuel Beck, Lasse Merz, Christoph Zorn, Leonel Merino, André van Hoorn, Fabian Beck: Can a Chatbot Support Software Engineers with Load Testing? Approach and Experiences. Under review 14 Chatbot-supported Performance Evaluation