Automated Testing Laboratory for Embedded Linux Distributions . - - PowerPoint PPT Presentation

▶
automated testing laboratory for embedded linux
SMART_READER_LITE
LIVE PREVIEW

Automated Testing Laboratory for Embedded Linux Distributions . - - PowerPoint PPT Presentation

Sep 04, 2022 134 likes •687 views

Automated Testing Laboratory for Embedded Linux Distributions . Pawe Wieczorek October 11, 2016 Samsung R&D Institute Poland Agenda . 1. Introduction 2. Motivation 3. Automation opportunities with our solutions 4. Future plans 5.

slide-1
SLIDE 1

Automated Testing Laboratory for Embedded Linux Distributions

.

Paweł Wieczorek October 11, 2016

Samsung R&D Institute Poland

slide-2
SLIDE 2

Agenda .

  • 1. Introduction
  • 2. Motivation
  • 3. Automation opportunities with our solutions
  • 4. Future plans
  • 5. Conclusion

1/42

slide-3
SLIDE 3

Introduction .

slide-4
SLIDE 4

Automated Testing Laboratory .

2/42

slide-5
SLIDE 5

Actual Automated Testing Laboratory .

3/42

slide-6
SLIDE 6

Automated Testing Laboratory – MinnowBoard Turbot .

4/42

slide-7
SLIDE 7

Automated Testing Laboratory – Odroid U3+ .

5/42

slide-8
SLIDE 8

Automated Testing Laboratory – HiKey .

6/42

slide-9
SLIDE 9

Automated Testing Laboratory – Supporting hardware .

7/42

slide-10
SLIDE 10

Automated Testing Laboratory – SD MUX .

8/42

slide-11
SLIDE 11

SD MUX .

9/42

slide-12
SLIDE 12

Motivation .

slide-13
SLIDE 13

Change life cycle .

10/42

slide-14
SLIDE 14

Change acceptance .

11/42

slide-15
SLIDE 15

Release engineering .

12/42

slide-16
SLIDE 16

Primary tools .

Open Build Service Jenkins

13/42

slide-17
SLIDE 17

Release Engineer role .

  • 1. Release engineer investigates build failures (if any)
  • 2. Release engineer checks whether new images introduce any regressions
  • 3. Release engineer approves inclusion of verified changes to the main

repository

14/42

slide-18
SLIDE 18

Release Engineer headache .

  • Complete image testing on multiple devices takes much time:

ttotal = tdownload + ntargets × (tflash + ttest)

  • Monotonous – involves repeating the same set of actions
  • Requires focus – processing similar results calls for an observant person

15/42

slide-19
SLIDE 19

Release Engineer dilemma .

  • 1. Can we test images less frequently?
  • 2. Can we run fewer tests on new images?
  • 3. Can we assume that successfully built packages work properly?

16/42

slide-20
SLIDE 20

Release Engineer credo .

  • 1. Resolve an issue as soon as it is discovered
  • 2. Look for a solution, not just workaround
  • 3. Don't release software that was never run on an actual device

17/42

slide-21
SLIDE 21

Room for improvement .

  • Complete image testing on multiple devices takes much time:

ttotal = tdownload + ntargets × (tflash + ttest)

  • Monotonous – involves repeating the same set of actions
  • Requires focus – processing similar results calls for an observant person

18/42

. . A U T O M A T E

slide-22
SLIDE 22

Automation opportunities with our solutions .

slide-23
SLIDE 23

Automation tasks categories .

  • Software
  • Infrastructure
  • Internal
  • External
  • Hardware

19/42

slide-24
SLIDE 24

Automation tasks examples .

  • Software
  • Infrastructure
  • Internal
  • External
  • Hardware
  • Polling OBS for new images
  • Getting new images from OBS
  • Controlling hosts and targets
  • Publishing test results
  • Flashing target devices with new images

20/42

slide-25
SLIDE 25

Automation tasks examples .

  • Software
  • Infrastructure
  • Internal
  • External
  • Hardware
  • Polling OBS for new images
  • Getting new images from OBS
  • Controlling hosts and targets
  • Publishing test results
  • Flashing target devices with new images

20/42

slide-26
SLIDE 26

Automation tasks examples .

  • Software
  • Infrastructure
  • Internal
  • External
  • Hardware
  • Polling OBS for new images
  • Getting new images from OBS
  • Controlling hosts and targets
  • Publishing test results
  • Flashing target devices with new images

20/42

slide-27
SLIDE 27

Automation tasks examples .

  • Software
  • Infrastructure
  • Internal
  • External
  • Hardware
  • Polling OBS for new images
  • Getting new images from OBS
  • Controlling hosts and targets
  • Publishing test results
  • Flashing target devices with new images

20/42

slide-28
SLIDE 28

Automation tasks examples .

  • Software
  • Infrastructure
  • Internal
  • External
  • Hardware
  • Polling OBS for new images
  • Getting new images from OBS
  • Controlling hosts and targets
  • Publishing test results
  • Flashing target devices with new images

20/42

slide-29
SLIDE 29

Software – polling OBS and getting new images .

  • OBS lacks event mechanism
  • Human-readable naming conventions

require parsing

  • New image discovery is run
  • n multiple levels
  • Scheduling tasks
  • Queueing tasks

Jenkins

21/42

slide-30
SLIDE 30

Internal infrastructure – reliable communication with devices .

OpenSSH

  • Depends on other services
  • Requires network connection

Serial console

  • Lower rate of data transfer
  • Less flexible than alternatives

Default choice

  • SDB

(Smart Development Bridge)

22/42

slide-31
SLIDE 31

Internal infrastructure – configuration management .

  • Testlab-handbook on its own is not enough
  • All changes in configuration are tracked in Testlab-host
  • Improved deployments
  • No more snowflakes!

23/42

slide-32
SLIDE 32

External infrastructure – results publishing .

  • Easily available
  • With possibility for future reuse
  • Preferably using existing services
  • Sharing test environment information
  • Publishing test results
  • Providing data for future reuse

MediaWiki edited by Pywikibot

24/42

slide-33
SLIDE 33

Hardware – flashing target devices with new images .

  • Current interface focused on user interaction
  • Designed for single target device per host
  • Architecture-specific procedure

25/42

slide-34
SLIDE 34

Hardware – SD MUX .

slide-35
SLIDE 35

Board control

Hardware – SD MUX .

slide-36
SLIDE 36

Board control Memory card

Hardware – SD MUX .

slide-37
SLIDE 37

Board control T arget SDB/card connection Memory card

Hardware – SD MUX .

slide-38
SLIDE 38

Board control Host card connection T arget SDB/card connection Memory card

Hardware – SD MUX .

slide-39
SLIDE 39

Board control Host SDB/card access Host card connection T arget SDB/card connection Memory card

Hardware – SD MUX .

slide-40
SLIDE 40

Power switch Board control Host SDB/card access Host card connection T arget SDB/card connection Memory card

Hardware – SD MUX .

slide-41
SLIDE 41

Controlling SD MUX .

$ sdmuxctrl --help Usage: sdmuxctrl command

  • l, --list
  • i, --info
  • o, --show-serial
  • r, --set-serial=STRING
  • t, --init
  • u, --status

(...)

33/42

slide-42
SLIDE 42

Former work flow .

Requires release engineer's interaction

34/42

slide-43
SLIDE 43

SD MUX work flow .

Fully automated process

35/42

slide-44
SLIDE 44

SD MUX – schematics .

slide-45
SLIDE 45

SD MUX – open-source .

https://git.tizen.org/cgit/tools/testlab/sd-mux

37/42

slide-46
SLIDE 46

Future plans .

slide-47
SLIDE 47

What is next? .

  • Pre-test cases development
  • More detailed monitoring of differences between tested images
  • Improved fail management
  • Improved resource management
  • System distribution

38/42

slide-48
SLIDE 48

Conclusion .

slide-49
SLIDE 49

Summary .

  • 1. No need for reinventing the wheel in modern automation
  • 2. Custom hardware can simplify tasks
  • 3. Automation pays off in the long term

39/42

slide-50
SLIDE 50

Questions?

39/42

slide-51
SLIDE 51

Thank you!

Paweł Wieczorek

p.wieczorek2@samsung.com Samsung R&D Institute Poland

slide-52
SLIDE 52

Further read

  • https://wiki.tizen.org/wiki/Laboratory
  • https://wiki.tizen.org/wiki/SD_MUX
  • https://git.tizen.org/cgit/tools/testlab
slide-53
SLIDE 53

Pictures used

  • https://wiki.tizen.org/w/images/9/95/Testlab.JPG
  • http://openbuildservice.org/images/obs-logo.png
  • https://wiki.jenkins-ci.org/download/attachments/2916393/logo.png
  • https://wiki.tizen.org/w/images/5/57/Tizen_Build_Process.gif
  • https://by-example.org/wp-content/uploads/2015/08/openssh-logo.png
  • https://pixabay.com/en/terminal-console-shell-cmd-dos-153150/
  • https://pixabay.com/en/gears-options-settings-silhouette-467261/
  • https://commons.wikimedia.org/wiki/File:Notification-icon-MediaWiki-logo.svg