AD30 - 5 Common Types of Mobile App Bugs Found Using - - PDF document

▶
ad30 5 common types of mobile app bugs found using ai
SMART_READER_LITE
LIVE PREVIEW

AD30 - 5 Common Types of Mobile App Bugs Found Using - - PDF document

Nov 21, 2023 448 likes •707 views

AD30 DevOps Practices 10:00 AM AD30 - 5 Common Types of Mobile App Bugs Found Using AI Presented by: Sandy

slide-1
SLIDE 1

¡ ¡ ¡ ¡ AD30 ¡

DevOps ¡Practices ¡ 10:00 ¡AM ¡ ¡ ¡ ¡ ¡ ¡

AD30 ¡-­‑ ¡5 ¡Common ¡Types ¡of ¡Mobile ¡App ¡ Bugs ¡Found ¡Using ¡AI ¡ ¡

Presented ¡by: ¡ ¡ ¡ ¡

Sandy ¡Park ¡

¡ ¡Apptest.ai ¡ ¡

Brought ¡to ¡you ¡by: ¡ ¡ ¡ ¡

¡

¡

¡ ¡

888-­‑-­‑-­‑268-­‑-­‑-­‑8770 ¡·√·√ ¡904-­‑-­‑-­‑278-­‑-­‑-­‑0524 ¡-­‑ ¡info@techwell.com ¡-­‑ ¡https://agiledevopswest.techwell.com/ ¡

¡

slide-2
SLIDE 2

¡

Sandy ¡Park ¡

¡ Sandy ¡Park ¡is ¡a ¡co-­‑founder ¡of ¡Apptest.ai, ¡an ¡emerging ¡startup ¡in ¡exploratory ¡mobile ¡ testing ¡located ¡in ¡the ¡bay ¡area ¡and ¡Seoul. ¡She ¡recently ¡automated ¡Facebook's ¡fiber ¡ connectivity ¡projects ¡while ¡partnering ¡with ¡MNOs ¡in ¡EMEA ¡and ¡APAC. ¡Prior ¡to ¡that, ¡she ¡ was ¡a ¡Director ¡of ¡mobile ¡partnerships ¡in ¡Samsung ¡USA. ¡She ¡also ¡contributed ¡to ¡the ¡ acquisition ¡of ¡EdgeSpring ¡to ¡Salesforce, ¡which ¡is ¡now ¡called ¡Salesforce ¡Wave. ¡She ¡holds ¡ a ¡Ph.D. ¡of ¡Energy ¡Resources ¡Engineering, ¡and ¡a ¡Master's ¡of ¡Electrical ¡Engineering ¡both ¡ from ¡Stanford ¡University. ¡

slide-3
SLIDE 3

4/29/19 1

5 Common Types of Mobile App Bugs Found Using AI

June 2019 Sandy Park

2

So Many Bugs, So Little Time

slide-4
SLIDE 4

4/29/19 2

App Errors, Overview What is AI Testbot 5 Types of Common Errors Principles Behind AI Testbot vs. Static Shift Left with CI Integrations

3

AGENDA

4

https://medium.com/@giltayar/not-only-cars-the-six-levels-of-autonomous-testing-a22611ab1d7f

AI TESTING AS MAJOR TREND

What’s it like to be L5 for testing? Gartner: “By 2022, at least 40% of new projects will have AI co-developers on the team.”

slide-5
SLIDE 5

4/29/19 3 50% Cost Saving Simple to Use 400% Increase in Release Frequency Test More! 300% Increase in Testing Capacity More Errors Caught 10% Increase in Customer Retention Higher Quality 10% Increase in Conversion Rate Less Poor Reviews!

5

BUSINESS IMPACT via AI TESTBOT

6

ERROR UNIVERSE from MACHINE

FOUND

Detectable by Logs

  • Crashes
  • Webpage not available

OTHERS

  • Context-specific
  • UI Aesthetics

We’ll come back later!

slide-6
SLIDE 6

4/29/19 4

7

CATEGORIZATION – PREVIOUS STUDIES

Sapienz: Multi-objective Automated Testing for Android Applications, ’12 Characterizing Android Specific Crash Bug, ‘19 Large-Scale Analysis of Framework-Specific Exceptions in Android Apps, ‘18 8

AI TESTBOT SUMMARY STATS

16,260 Test Runs 1,107 Apps Tested 214 Buggy Apps (19.3%)

Broken URL Link 25% Native 13% NullPointerException 9% IndexOutOfBoundsException 9% FileUriExposedException 5% Resource Limitation Error 4% InflateException 4% IllegalStateException 3% Out of memory 3% UnsatisfiedLinkError 3% BadTokenException 2% RuntimeException 2% ETC 18%

slide-7
SLIDE 7

4/29/19 5

9

PERFORMANCE METRIC 1 - COVERAGE

PERFORMANCE METRIC 2 – ERROR HIT RATE

10

Applications Actions Unique Screenshots Result Actions Unique Screenshots Result

AccuWeather 214 97 Error(s) Found 116 65 Clean/No Errors MLB At Bat 269 119 Error(s) Found 12 11 Clean/No Errors Bitmoji 301 118 Error(s) Found 16 12 Clean/No Errors The Honest Co 360 90 Error(s) Found 98 44 Error(s) Found Smiley Emoji Keyboard 2018 317 79 Error(s) Found 17 11 Clean/No Errors BK Booster 72 34 Error(s) Found 34 24 Error(s) Found AW Security Cleaner 79 19 Error(s) Found 46 29 Clean/No Errors NBC Sports Scores 275 125 Error(s) Found 48 45 Clean/No Errors News Break: Local & Breaking 302 135 Error(s) Found 20 15 Clean/No Errors Pinterest 293 31 Error(s) Found 16 8 Clean/No Errors Super Speed - Clean & Booster 160 28 Error(s) Found 23 18 Clean/No Errors Tubi TV - Free Movies & TV 303 146 Error(s) Found 95 28 Clean/No Errors King James Bible 191 124 Error(s) Found 247 224 Clean/No Errors Reddit 309 176 Error(s) Found 171 71 Clean/No Errors CNBC 142 49 Error(s) Found 18 12 Clean/No Errors

Apptest.ai Detected More Errors 15 vs. 2 among US Top 100 Apps (Q2 ‘18)

slide-8
SLIDE 8

4/29/19 6

11

HOW IT WORKS

Deep Learning NLP NER Ontology UI Detection Virtual Persona Reinforcement Learning

12

Step1: Object Detection

Text 1 Option A Option B Button 1 Button 2 Text A Text B Image 1 Step2: Layout – Grouping and Labelling Image: Image 1 Button1 : “Play” Button2 : “Save”

Option A : Text A Option B: Text B

Group 2 Group 3 Text: Text 1 Group 1

Group 2 Group 1 Group 3

Type: button Label: “Play” Order: 1 Value: N/A Type: button Label: “Save” Order: 1 Value: N/A Type: Option Label: Text A Order: 1 Value: Unchecked Type: Option Label: Text B Order: 2 Value: Unchecked Type: Image Label: Image 1 Order: 1 Value: N/A Type: Text Label: Text 1 Order: 2 Value: N/A

Step3: Action Path

slide-9
SLIDE 9

4/29/19 7

PRODUCT SNAPSHOT

14

COMMON ERROR TYPES

Out of 39,000+ tests & 3,300+ bugs

Broken Link Null Pointer Exception Index Out of Bounds Exception Out of Memory Bad Token Exception

slide-10
SLIDE 10

4/29/19 8

15

Broken Link 1

Tubi TV 2.12.9 / LG Nexus 5 / Android 6.0

16

Broken Link 2

MLB At Bat 7.5.0 / Google Pixel / Android 9.0

slide-11
SLIDE 11

4/29/19 9

17

Null Point Exception 1

Auto Reply for WA 7.5.3 / LG Nexus 5 / Android 6.0

18

Heart Radio 4.10.14 / LG Nexus 5X / Android 8.0

Null Point Exception 2

slide-12
SLIDE 12

4/29/19 10

19

Index Out of Bounds 1

Lyft 5.23.3 / LG Nexus 5 / Android 6.0

20

Index Out of Bounds 2

DuoLingo 5.23.3 / LG Nexus 5 / Android 6.0

slide-13
SLIDE 13

4/29/19 11

21

Out of Memory 1

End Game Wallpaper 1.0 / Samsung Galaxy S5 / Android 6.0

22

Out of Memory 2

Emoji Keyboard 1.1.7 / Samsung Galaxy S7 Edge / Android 7.0

slide-14
SLIDE 14

4/29/19 12

23

Bad Token Exception 1

UAN e-Sewa 1.1 / Samsung Galaxy S7 Edge / Android 7.0

24

Bad Token Exception 2

Snake Screen Joke 1.0 / LG G6 / Android 8.0

slide-15
SLIDE 15

4/29/19 13

25

Dynamic Testing vs. Static Testing

26

Unidentified Resource IDs and Descriptions → Lack of information to automate

Limitation on Code Based Automation 1

slide-16
SLIDE 16

4/29/19 14

27

Incorrect Clickable Properties

Limitation on Code Based Automation 2

28

Multi-layer Detection

Former technology remains taking actions on background which is irrelevant.

slide-17
SLIDE 17

4/29/19 15

29

(1) (2) (3) (4) (1) (2) (3) (4) (1) (2) (3) (4) (5) (6)

Sequencing Correct Input Order

(5)

30

Input Relevant Value Strings

Random string for input slots Correct input format detection for location, number, profile data, etc.

slide-18
SLIDE 18

4/29/19 16

31

Mobile UI Recognition

Deep Learning and Training for UI Element recognition

32

Named Entity Recognition

Predicting the login based on certain patterns and using relevant name as an input for search.

https://www.analyticsvidhya.com/blog/2019/02/flair-nlp-library-python/

slide-19
SLIDE 19

4/29/19 17

33

My app is updated every day. Can I test my app every day?

Hell No! ← Manual Testing Maybe. ← QA Scripting

Yes! ←

34

Inconsistent and time consuming experience CODE BUILD MANUAL TEST MANUAL TEST MANUAL TEST MANUAL TEST MANUAL TEST MANUAL TEST MANUAL TEST DEPLOY

Conventional MANUAL Functional Testing

slide-20
SLIDE 20

4/29/19 18

35

Time consuming and requires learning tools or expertise

Conventional AUTOMATED Functional Testing

CODE BUILD CODE TEST DEPLOY TEST

36

CODE BUILD CODE TEST DEPLOY TEST

Codeless, AI based, continuous automated testing. Zero coding and knowledge required. Detect issues much earlier in the pipeline.

Continuous Testing with AI TESTBOT

Test More! Sync with your code update

slide-21
SLIDE 21

4/29/19 19

37

Integrate with your Favorite CI Tool

38

Jenkins with AI Testbot

slide-22
SLIDE 22

4/29/19 20

39

Configure CI Pipeline (Jenkins)

40

Access Test Results (Jenkins)

slide-23
SLIDE 23

4/29/19 21

  • Just Add 5 lines to the configuration.

Circle CI with AI Testbot

  • rbs:

testbot: saucelabs/ios-app-test@volatile …

  • testbot/run-test::

binary_path: app/build/outputs/apk/app-debug.apk project_id: '818‘ …

.circleci/config.yml

  • Just Add 4 lines to the configuration.

Bitrise with AI Testbot

…

  • saucelabs-testbot:

inputs:

  • binary_path: HackerNews.ipa
  • project_id: 825

…

bitrise.yml

slide-24
SLIDE 24

4/29/19 22

  • Just Add 7 lines to the configuration.

Microsoft Azure with AI Testbot

…

  • bash: |

git clone https://github.com/apptestai/ci_addon export binary_path=HackerNews.ipa export project_id='831' bash ./ci_addon/step.sh displayName: 'Tested by TestBot' env: APPTEST_AI_ACCESS_KEY: $(APPTEST_AI_ACCESS_KEY)…

azure-pipelines.yml

For the last eight months, NBC Owned Television Station was a pioneer to run its AI Bot to test their Android and iOS apps in

  • production. Now it runs in all stages, everyday!
slide-25
SLIDE 25

4/29/19 23

Confidential

THANK YOU