SLIDE 1
Nadia Alshahwan
FACEBOOK SOFTWARE ENGINEER SAPIENZ TEAM; THIS TALK IS BASED ON WORK OF THE WHOLE TEAM
Automated Test Design and Bug Fixing @ Facebook
Automated Test Design and Bug Fixing @ Facebook Nadia Alshahwan - - PowerPoint PPT Presentation
Automated Test Design and Bug Fixing @ Facebook Nadia Alshahwan - - PowerPoint PPT Presentation
Automated Test Design and Bug Fixing @ Facebook Nadia Alshahwan FACEBOOK SOFTWARE ENGINEER SAPIENZ TEAM; THIS TALK IS BASED ON WORK OF THE WHOLE TEAM Unfortunately, your app has stopped. Engineers Design Machines Execute System Level
SLIDE 2
SLIDE 3
Unfortunately, your app has stopped.
SLIDE 4
Engineers Design Machines Execute
SLIDE 5
System Level UI Testing
RANDOM FUZZER
SLIDE 6
System Level UI Testing
RANDOM FUZZER HUMAN TESTERS
SLIDE 7
System Level UI Testing
RANDOM FUZZER HUMAN TESTERS
SLIDE 8
What is Sapienz?
SLIDE 9
IN PROC. OF ISSTA'16, 2016.
Sapienz: Multi-Objective Automated Testing for Android Applications
KE MAO, MARK HARMAN, AND YUE JIA
SLIDE 10
Test live in a search space
SLIDE 11
Auto Test Design Auto Fix Auto Boost Auto Test Design
SLIDE 12
Auto Test Design
Auto Fix Auto Boost Auto Test Design
SLIDE 13
Features
GENETIC ALGORITHM MULTI- OBJECTIVE MOTIF INTERACTION
SLIDE 14
LOGIN USERNAME PASSWORD
SLIDE 15
TEST GENERATION
SLIDE 16
TEST GENERATION SELECTION CROSSOVER
SLIDE 17
TEST GENERATION CROSSOVER MUTATION
SLIDE 18
TEST GENERATION MUTATION
SLIDE 19
COVERAGE SHORTEST POSSIBLE SEQUENCE
COVERAGE LENGTH PARETO FRONT
SLIDE 20
Smarter CI
SLIDE 21
REOCCURRING OPERATOR
Sapienz for Smarter CI
RELEASE BUILD CODE COLLABORATION ENGINEER REVIEWER AUTO TRIAGE AUTO TEST DESIGN AUTO FIX CONTINUOUS BUILD
SLIDE 22
REOCCURRING OPERATOR
Sapienz Workflow
OPERATIONS IN THE EVOLUTION RUN WORKFLOW BUILD SEARCH CRASH TRIAGE MOBILE BUILD DB ENGINE
SLIDE 23
Fault Triage Process
OPERATIONS IN THE EVOLUTION WORKFLOW TRIAGE BUILD SEARCH CRASH REOCCURRING OPERATOR FILE A FIX STACK TRACE
A B C D
DIFF OWNER REVIEW LOCALIZER CRASH DATA
SLIDE 24
POST-LAND BUILD SMOKE BUILD
Diff-time Signals
SLIDE 25
Test Design
ASYNCIO
PROVIDER
HOST
PYTHON 3
EVOLUTION ENGINE CONTROLLER ANALYZER PROXY FBLEARNER IO CLIENT MOTIFCORE
SLIDE 26
~75%
FIX RATE
SLIDE 27
Distribution (FB)
Null Pointer Illegal State Illegal Argument RunTime Class Cast No Such Method Assertion Array Index Out of Bounds Bad Token Index Out of Bounds TOP CRASHES TYPES ON FACEBOOK FOR ANDROID (BY SAPIENZ)
SLIDE 28
Distribution (Research)
TOP CRASHES TYPES ON 1000 ANDROID APPS (BY SAPIENZ[1]) Null Pointer Illegal State Illegal Argument RunTime Activity Not Found Out of Memory Concurrent Array Index Out of Bounds Bad Token Index Out of Bounds
[1] K. MAO, M. HARMAN, AND Y. JIA, “SAPIENZ: MULTI-OBJECTIVE AUTOMATED TESTING FOR ANDROID APPLICATIONS,” IN PROC. OF ISSTA’16, 2016 SLIDE 29
Support on Facebook App Family
INSTAGRAM FACEBOOK MESSENGER WORKPLACE FBLITE
SLIDE 30 Workplac e for Android Messenge r for Android Instagram for Android Facebook for iOS Facebook for Android
SLIDE 31
Auto Fix
Auto Boost Auto Test Design Auto Fix
SLIDE 32
CRASH TRIGGERS
Auto Fix Workflow (Generation)
REVERT FULL DIFF REVERT PARTIAL DIFF TEMPLATE MUTATION SAPIENZ AUTO TRIAGE FIX PATCH GENERATOR TRIGGER PATCH GENERATOR CREATE REVISION IN SAPFIX
SLIDE 33
Auto Fix Workflow (Generation)
CRASH TRIGGERS REVERT FULL DIFF REVERT PARTIAL DIFF TEMPLATE MUTATION SAPIENZ AUTO TRIAGE FIX PATCH GENERATOR TRIGGER PATCH GENERATOR CREATE REVISION IN SAPFIX
SLIDE 34
Auto Fix Workflow (Validation)
SLIDE 35
Auto Fix Workflow (Validation)
CREATE REVISION IN SAPFIX PATCH 1 PATCH 2 PATCH N MANUALLY WRITTEN TESTS COMPILATION ERROR? BUILD CRASH FIXED? NEW CRASH? SAPIENZ E2E
SLIDE 36
Auto Fix Workflow (Validation)
CREATE REVISION IN SAPFIX PATCH 1 PATCH 2 PATCH N MANUALLY WRITTEN TESTS COMPILATION ERROR? BUILD CRASH FIXED? NEW CRASH? SAPIENZ E2E
SLIDE 37
Auto Fix Workflow (Signal)
CREATE REVISION IN SAPFIX
REJECTED/ EXPRED
DEVELOPER REVIEWS AUTO FIX PROCESSOR DEVELOPER INTERACTION TRACKER
ACCEPTED
SLIDE 38
Auto Boost Auto Test Design Auto Fix Auto Boost
SLIDE 39
Auto Boost
Auto Test Design Auto Fix Auto Boost
SLIDE 40
AFFECTED USER PREDICTION PROD CRASH LINK INFER LINK
Signal boosting
SLIDE 41
IN PROC. OF SSBSE, 2018.
Deploying Search Based Software Engineering with Sapienz at Facebook
NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO, ALEXANDER MOLS, TAIJIN TEI AND ILYA ZORIN
Resources
SLIDE 42
Flaky tests Fix detection Automated Oracles Smarter white box coverage Combining static and dynamic Unit test from system tests Human machine test hybrids Fully parallel search algorithms Auto Fix and perf improvement
SLIDE 43
Nadia Alshahwan
nalshahwan@fb.com