PUMA: Programmable UI Automation for Large-Scale Dynamic Analysis of - PowerPoint PPT Presentation

PUMA: Programmable UI Automation for Large-Scale Dynamic Analysis of Mobile Apps Shuai Hao, Bin Liu, Suman Nath, William G.J. Halfond, Ramesh Govindan

2 Mobile App Explosion 1.2 million 1,200,000 1,000,000 Number of Apps 800,000 600,000 400,000 200,000 0 Source: http://en.wikipedia.org/wiki/Google_Play

3 App Behaviors are Complex Source: forbes.com Source: droidforums.net Source: zdnet.com

4 Dynamic Analysis These app behaviors can be studied by dynamic analysis. But given the huge number of apps, We need scalable dynamic analysis methods.

Dynamic Analysis with UI Automation 5  UI-driven app exploration (or Monkey) Source: http://mttnow.com/ios-automated-ui-testing

6 Monkey: High-level Idea UI events UI info Monkey

7 Monkey in Dynamic Analysis Example: Check number of Ads on each page

8 Monkey: Overview Page Text List Image UI Extraction UI Structure Button Button UI Extraction Channel Monkey UI Action Channel Decide UI Action Action Dictionary Click “Continue” Button Button Click UI Action Multipage Scroll List Scroll The set of previously Next App Page An App Page visited pages

9 Monkey-based App Analyses AMC Check accessibility of UI elements MobiSys’13 DECAF Check ad violation on each page NSDI’14 SmartAds Deliver ad based on page content MobiSys’13 VanarSena Test whether app crashes MobiSys’14 ContextualFuzzing Test app crash and performance issue MSR- TR’13 AppsPlayground Detect privacy leak and malware CODASPY’13

10 Monkey Coverage vs. Speed Problem Monkey may never finish app exploration Solution Optimize app exploration in an analysis-specific way Refresh Example Updates can also be AMC: uses a structural similarity measure pushed to device DECAF: uses a structural similarity measure and ML- based technique to avoid similar page visit

11 Flexibility in Access to App Info Problem Analysis may require data that is not available in UI structure Solution Instrument app or extract info from environment Example SmartAds: uses app instrumentation to obtain in- app context info VanarSena: uses app instrumentation to control memory and isolated storage

12 Monkey-based App Analyses Result Complex and customized code e.g. DECAF: > 4300 LOC with 70% for monkey Impact Tool for analysis X is very hard to be reused for analysis Y.

13 Our Goal Provide a programmable way for monkey-based app analyses. • so that both analyses X and Y can be implemented easily by customization

14 Approach Event-Driven UI Automation UI Property Ad Fraud Page Content Monkey Monkey Monkey AMC DECAF SmartAds

15 Approach Event-Driven UI Automation Ad Fraud UI Property Page Content Ad Fraud Monkey UI Property Monkey Page Content Monkey DECAF AMC SmartAds events events events API Programmable PUMA Monkey

16 Framework Requirements Exploration Page Property Action Required System Transition Target Checked Taken Instrumentation AMC Distinct UI Accessibility - No DECAF Distinct UI Ad layout - No Page SmartAds All UI - Yes content 3 A E Distinct UI None - Yes Apps Distinct UI, text Info flow - Yes Playground Inject VanarSena Distinct UI, text Crash Yes fault Contextual Change All UI Crash, perf . No Fuzzing context Customize the definition of state equivalence Customize the definition of next state Specify instrumentation for the app Specify which properties to check Change the environment

17 Key Idea: Event-driven Paradigm

18 Key Idea: Event-driven Paradigm App-specific Events Go to initial page Next Click Pick a clickable UI element Effect environment changes, if needed Modify environment Perform click, user input Text Input Wait for page done In-line Analysis Check page equivalence State Equivalence Need continue Terminating App

19 Framework Requirements Exploration Page Property Action Required System Transition Target Checked Taken Instrumentation AMC Distinct UI Accessibility - No DECAF Distinct UI Ad layout - No Modify environment Page SmartAds State Equivalence All UI - Yes content 3 A E Distinct UI None - Yes Apps Distinct UI, text Info flow App-specific Events - Yes Playground Inject VanarSena Distinct UI, text Crash Yes fault Contextual Change All UI Crash, perf . No Fuzzing context Text Input Next Click In-line Analysis

20 Example: Network Usage Profiler Count bytes sent and received in HTTP traffic 400 KB State Equivalence 1 MB Next Click <described later> 200 KB App-specific Events 500 KB

21 Example: Network Usage Profiler Count bytes sent and received in HTTP traffic State Equivalence <described later>

22 State Equivalence Determine whether two pages are equivalent or not Structural similarity, content similarity, customized

23 Example: Network Usage Profiler Count bytes sent and received in HTTP traffic Next Click <described later>

24 Next Click Determine which UI element to click Sequential order, max-type order, customized

25 Example: Network Usage Profiler Count bytes sent and received in HTTP traffic … App-specific Events

26 App-Specific Events Instrumentation through SIF  regexp-based filter language  Users define named event  Associate named event with a codepoint in the binary S. Hao, D. Li, W. Halfond, R. Govindan . “SIF: Selective Instrumentation Framework for Mobile Applications.” In ACM MobiSys , 2013.

27 Putting It All Together Original PUMAScript App Instrumented Instrumenter App Interpreter Monkey PUMA

28 Implementation Challenges  Obtaining UI structure  Intercept uiautomator events  Supporting page scrolling  Check for scrolling direction  Scroll in a zig-zag pattern  Combine partial pages  Detecting page loading completion  WINDOW_CONTENT_CHANGED event  Use window-based moving average heuristic

29 Evaluation Research Questions Can PUMA support rapid development of 1 large-scale dynamic app analyses? What insights into the app ecosystem can 2 these analyses provide?

30 Evaluation Methodology  Implemented 7 analysis studies  Downloaded 3,600 apps from Google Play  Ran over emulators phones

31 Implemented Analysis Studies Accessibility Check accessibility of UI elements Violation Detection Content-based Crawl in-app data for search engine App Search UI Structure Classify apps based on UI structure Classifier Ad Fraud Detect ads placement violation Detection Network Usage Profile in-app HTTP traffic use Profiler Permission Usage Profile in-app permission usage Profiler Stress Testing Inject null HTTP response for test

32 RQ1: Rapid Development? Accessibility Violation Detection Content-based App Search UI Structure Finished development Classifier and experiments within Ad Fraud Detection 2 weeks Network Usage Profiler Permission Usage Profiler Stress Testing

33 RQ1: Rapid Development? PUMAScript User Code Study Task (LOC) (LOC) Accessibility violation detection 11 60 Content-based app search ~4300 LOC 14 0 UI structure classifier 11 0 Ad fraud detection 11 52 Network usage profiler 19 8 Permission usage profiler 20 5 Stress testing 16 5 PUMA only uses < 100 LOC for all studies

34 Cross-Analysis Scaling Optimization With PUMA Computing Resources UI Property Monkey API Ad Fraud Monkey API Page Content Monkey API

35 Cross-Analysis Scaling Optimization With PUMA after page is loaded UI Property Monkey API check UI property after page is loaded 1. check UI property For 100 apps and 1 phone, it took 8 hours after page is loaded Ad Fraud 2. check Ad fraud instead of 20 hours to finish. Monkey API check ad fraud 3. crawl page content  2.5x speed up after page is loaded Page Content Monkey API crawl page content

36 RQ2: Insights into App Ecosystem? Accessibility Accessibility Check accessibility of UI elements Check accessibility of UI elements Violation Detection Violation Detection Content-based Content-based Crawl in-app data for search engine Crawl in-app data for search engine App Search App Search UI Structure Classify apps based on UI structure Classifier Ad Fraud Detect ads placement violation Detection Network Usage Profile in-app HTTP traffic use Profiler Permission Usage Profile in-app permission usage Profiler Stress Testing Inject null HTTP response for test

37 RQ2: Insights into App Ecosystem? Accessibility Violation Detection Start Stop Start Start Start Stop Button Button Scrolling Distance Size Word Count K. Lee, J. Flinn, T. J. Giuli, B. Noble, C. Peplin. “AMC : Verifying User Interface Properties for Vehicular Applications.” In ACM MobiSys’13.

38 RQ2: Insights into App Ecosystem? Accessibility Violation Detection scrolling button distance button size word count user action per task 0 500 1000 1500 2000 2500 ~55% of our apps violate ≥1 accessibility rules

39 RQ2: Insights into App Ecosystem? Content-based App Search