Automated Test Design and Bug Fixing @ Facebook Nadia Alshahwan - PowerPoint PPT Presentation

Automated Test Design and Bug Fixing @ Facebook Nadia Alshahwan FACEBOOK 
 SOFTWARE ENGINEER SAPIENZ TEAM; THIS TALK IS BASED ON WORK OF THE WHOLE TEAM

Unfortunately, 
 your app has stopped.

Engineers Design Machines Execute

System Level UI Testing RANDOM FUZZER

System Level UI Testing RANDOM FUZZER HUMAN TESTERS

System Level UI Testing RANDOM FUZZER HUMAN TESTERS

What is Sapienz?

Sapienz: Multi-Objective Automated Testing for Android Applications KE MAO, MARK HARMAN, AND YUE JIA IN PROC. OF ISSTA'16, 2016.

Test live in a search space

Auto Test Design Auto Test Design Auto Fix Auto Boost

Auto Test Design Auto Fix Auto Boost Auto Test Design

Features MOTIF GENETIC MULTI- ALGORITHM OBJECTIVE INTERACTION

USERNAME PASSWORD LOGIN

TEST GENERATION

TEST GENERATION CROSSOVER SELECTION

TEST GENERATION CROSSOVER MUTATION

TEST GENERATION MUTATION

COVERAGE PARETO FRONT LENGTH COVERAGE SHORTEST POSSIBLE SEQUENCE

Smarter CI

Sapienz for Smarter CI REVIEWER AUTO FIX ENGINEER CODE COLLABORATION CONTINUOUS RELEASE BUILD BUILD AUTO TRIAGE AUTO TEST DESIGN REOCCURRING OPERATOR

Sapienz Workflow MOBILE BUILD ENGINE DB BUILD SEARCH CRASH TRIAGE REOCCURRING OPERATOR OPERATIONS IN THE EVOLUTION RUN WORKFLOW

Fault Triage Process OPERATIONS IN THE EVOLUTION WORKFLOW BUILD SEARCH CRASH TRIAGE REOCCURRING 
 OPERATOR A B C OWNER DIFF REVIEW CRASH DATA LOCALIZER D STACK TRACE FILE A FIX

Diff-time Signals POST-LAND BUILD SMOKE BUILD

Test Design PYTHON 3 PROVIDER CONTROLLER EVOLUTION 
 ENGINE ANALYZER ASYNCIO PROXY FBLEARNER IO MOTIFCORE HOST CLIENT

~75% FIX RATE

Distribution (FB) Null Pointer Illegal State Illegal Argument RunTime Class Cast No Such Method Assertion Array Index Out of Bounds Bad Token Index Out of Bounds TOP CRASHES TYPES ON FACEBOOK FOR ANDROID (BY SAPIENZ)

Distribution (Research) Null Pointer Illegal State Illegal Argument RunTime Activity Not Found Out of Memory Concurrent Array Index Out of Bounds Bad Token Index Out of Bounds TOP CRASHES TYPES ON 1000 ANDROID APPS (BY SAPIENZ [1] ) [1] K. MAO, M. HARMAN, AND Y. JIA, “SAPIENZ: MULTI-OBJECTIVE AUTOMATED TESTING FOR ANDROID APPLICATIONS,” IN PROC. OF ISSTA’16, 2016

Support on Facebook App Family WORKPLACE INSTAGRAM FACEBOOK FBLITE MESSENGER

Facebook for iOS Facebook for Android Workplac e for Android Messenge r for Android Instagram for Android

Auto Test Design Auto Fix Auto Boost Auto Fix

Auto Fix Workflow (Generation) CRASH TRIGGERS SAPIENZ AUTO 
 TRIGGER PATCH FIX PATCH CREATE REVISION TRIAGE GENERATOR GENERATOR IN SAPFIX REVERT 
 REVERT 
 TEMPLATE MUTATION FULL DIFF PARTIAL DIFF

Auto Fix Workflow (Generation) CRASH TRIGGERS SAPIENZ AUTO 
 TRIGGER PATCH FIX PATCH CREATE REVISION TRIAGE GENERATOR GENERATOR IN SAPFIX REVERT 
 REVERT 
 TEMPLATE MUTATION FULL DIFF PARTIAL DIFF

Auto Fix Workflow (Validation)

Auto Fix Workflow (Validation) PATCH 1 CREATE REVISION IN SAPFIX MANUALLY PATCH 2 WRITTEN TESTS PATCH N BUILD SAPIENZ E2E COMPILATION ERROR? CRASH FIXED? NEW CRASH?

Auto Fix Workflow (Validation) PATCH 1 CREATE REVISION IN SAPFIX MANUALLY PATCH 2 WRITTEN TESTS PATCH N BUILD SAPIENZ E2E COMPILATION ERROR? CRASH FIXED? NEW CRASH?

Auto Fix Workflow (Signal) ACCEPTED CREATE REVISION AUTO FIX DEVELOPER DEVELOPER IN SAPFIX PROCESSOR INTERACTION REVIEWS TRACKER REJECTED/ EXPRED

Auto Test Design Auto Fix Auto Boost Auto Boost

Auto Test Design Auto Fix Auto Boost Auto Boost

Signal boosting PROD CRASH LINK AFFECTED USER PREDICTION INFER LINK

Resources Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO, ALEXANDER MOLS, TAIJIN TEI AND ILYA ZORIN IN PROC. OF SSBSE, 2018.

Flaky tests Fix detection Automated Oracles Smarter white box coverage Combining static and dynamic Unit test from system tests Human machine test hybrids Fully parallel search algorithms Auto Fix and perf improvement

Nadia Alshahwan nalshahwan@fb.com