Embedded Software TI2726-B 8. Debugging techniques Koen Langendoen - - PowerPoint PPT Presentation

Embedded Software

Koen Langendoen

Embedded Software Group

TI2726-B

• 8. Debugging techniques

2

Grace Hopper 1947

Overview

 Debugging techniques  Debugging a distributed system  Preparation for the exam

Debugging

 Write good code! Testing only uncovers a fraction of bugs  Testing and debugging can be very difficult

 HW is involved (radio communication over devices?)

 Different perception on bugs for ES

 MS IE crashed?  Elevator not working? Telephone calling the wrong person?

Cash machine? Medical instruments?

4

Testing on the host machine

 Try finding bugs early in the development process

 HW might not be available early on

 Exercise all of the code

 Unlikely situations difficult to test (events on two devices)

 Develop reusable, repeatable tests  Leave an “audit trail” of the test results

 usually ES do not have hard-drives

 Conclusion: do not test on target more than needed!

5

6 | 40

Coverage tools

$ gcc -fprofile-arcs -ftest-coverage tmp.c $ a.out Success $ gcov tmp.c 87.50% of 8 source lines executed in file ‘tmp.c’ Creating ‘tmp.c.gcov’

gcov

• : 1:#include <stdio.h>
• : 2:

1: 3:main () {

• : 4: int i, total;

1: 5: total = 0; 11: 6: for (i = 0; i < 10; i++) 10: 7: total += i; 1: 8: if (total != 45) #####: 9: printf ("Failure\n");

• : 10: else

1: 11: printf ("Success\n"); 1: 12:}

• : 1:#include <stdio.h>
• : 2:

1: 3:main () {

• : 4: int i, total;

1: 5: total = 0; 11: 6: for (i = 0; i < 10; i++) 10: 7: total += i; 1: 8: if (total != 45) #####: 9: printf ("Failure\n");

• : 10: else

1: 11: printf ("Success\n"); 1: 12:}

7 | 40

Coverage tools

www.ECLEmma.org

• Eclipse

plugin for Java

Testing on the host machine

8

Hardware independent code Hardware dependent code Hardware

Target system

Hardware independent code Test scaffold code Keyboard Display Disk

Test system

Testing on the host machine

 Divide code into two categories

 Hardware-dependent code  Hardware-independent code

 Scaffold code provides the same entry points as the HW-

dependent code

 HW-dependent code can be debugged on HW only

 Replace calls to HW functions with meaningful information

 Rather than replacing the function in() for radio, replace the layer

above with a printf()

9

Testing on the host machine

 Calling interrupt routines

 Interrupt routines are major part of the system  Calling interrupt routines needs to be done from scaffold

 Difficult? Not really  Split code of the ISR in two: HW-dependent and HW-independent  Place the HW-independent code in a separate function  Test this code extensively

 Example: character processing on the serial line

10

Calling the timer interrupt

 Timer ISR – one of the most used and important parts of

the system

 Alternatives:

 Emulate HW to “naturally” call this ISR  Force calls from the test scaffolds

 Second option is preferred

 The overhead is not large on a PC system  You have control in deciding when the time ISR is called

with respect to other interrupts

 Have several events occur “at the same time”

11

Script files

 Automation is needed almost always

 Validate a piece of code under the same input conditions  “Trigger” ISRs in the same manner each time

 Using script files requires some small overhead

 Parser needed

 Very simple instructions (2-3 characters)  Comments must be allowed  Data has to be entered ASCII & Hexadecimal

 Results reporting in a formatted form

12

Script files - interleaving

13

# Frame arrives (beacon with no element) # Dst Src Ctrl Typ Stn Timestamp mr/56 ab 0123456789ab 30 00 6a6a # backoff time expires (software should send frame) kt0 # timeout expires again (assoc fails, no response) kt0 #Some time passes (software should retry sending) kn2 kn2 # another beacon arrives # Dst Src Ctrl Typ Stn Timestamp mr/56 ab 0123456789ab 30 00 6a6a # Frame arrives (beacon with no element) # Dst Src Ctrl Typ Stn Timestamp mr/56 ab 0123456789ab 30 00 6a6a # backoff time expires (software should send frame) kt0

• -> Sending frame: ab ff ab ...

# timeout expires again (assoc fails, no response) kt0 #Some time passes (software should retry sending) kn2 kn2

• -> Sending frame ab ff ab ...

# another beacon arrives # Dst Src Ctrl Typ Stn Timestamp mr/56 ab 0123456789ab 30 00 6a6a

Input script Output file

More advanced scripting

 Automate state machine behavior

 Each time some actuation takes place, a certain input (ISR)

should be triggered

 This behavior is an alternative to the regular behavior

 Emulate the communication medium between devices

 Wireless communication – interference, multiple devices

 barcode reader, any wireless interface…

14

Test scaffold software = communication medium Scanner A Scanner B Register A Register B

Objections!!!

 Objections to debugging on host machine appear

 Engineer fails to quantify correctly the needed effort  Boss wants code to be ready fast  Everyone fails to realize how much time debugging will take

 Most common objection

 Testing on host is useless: most of code is HW-dependent

15

Example from Telegraph

16

Objections!!!

 Building a test scaffold is difficult misconception

 Writing a simple parser is easy even if done in C  Code output capture is a simple formatting function  Debugging takes significantly longer than code writing

 RTOS needs to run on the host system

 If so, choose one RTOS that does (many versions available)  If not, emulate system calls! Use a shell around the OS

17

Objections!!!

 Other objections

 Software interaction with hardware

 Example: use the wrong address to access the UART

 Response and throughput  Shared-data problems (assembly level)  Portability issues (big-endian vs. little-endian)

 They are true!

 You cannot (usually) test these on the host  Testing on the host is not aimed at these issues

18

Instruction set simulators

 The processor execution is emulated in software

 Software executing assembly version of real code

 Useful for testing some of the issues not covered by

debugging on the host

 Advantages

 Determine response time and throughput  Testing assembly-language routines  Resolving portability issues  Testing code dealing with peripherals

 What they cannot do

 Shared-data bugs – unless scripting is allowed or you are lucky  Simulate the whole platform

19

The ASSERT macro

 Very used tool:

assert(condition);

 If condition is true, noting happens  If condition is false, user code is launched

 Print a message and stop

 Advantages

 Program state is tested on the spot: localized bugs  Example usage:

 Test input values of a function (passing NULL pointers?)  Test context of function calls  assert(!” bad return from function”)

20

The ASSERT macro

 Assert on the target platform

 Less verbose as systems do not usually have displays  Redefine functionality of assert

 Disable interrupts and do: “while(1){};”  Turn on LEDs in a given pattern (led-based debugging)  Write variables to a certain location of memory  Write the location of the current instruction  Execute illegal instruction to trigger debugger or stop

21

Using lab tools

 Multimeter

 Measure voltage between various points

 Use it to detect if circuits are powered  Test state of enable/disable pins

 Measure resistance between points

 Make sure device is off – detect short or open circuits

 Oscilloscope

 Software engineers usage (time, voltage, trigger)

 Use it as a voltmeter  Check if circuit is working at all by looking at changing waves  Check if signals are changing as expected (communication lines)  Check timing issues

22

Logic analyzer

 A “must-have” tool for embedded software design

 Usb-based cheap versions available

 Can track a set of digital signals simultaneously

 Only VCC and ground signals understood  Logic analyzers are storage devices  Significantly more complex triggering

mechanisms

 Timing mode (make use of “real” time)

 Test if event ever occurs  Measure time with respect to code length

and response

 Control output signal patterns

23

Logic analyzer

 State mode (makes use of external time signals)

 Use the RE signal on memories to monitor memory access  Trace of execution display: signal, hex value, assembly code, line nr.  Trigger on a special event (memory access) and go backwards in time  Trigger writing of bad values in RAM (NULL pointers as parameters)  Filters on the data values are available

24

Using lab tools

 In circuit emulators

 Hardware devices emulating the whole processor  Allow breakpoints and memory/registry inspection  Desktop debugger + logic analyzer in one tool  Logic analyzers are preferred: any processor, timing mode,

better filters, easy to install

 Monitors

 Two parts: on the target system and on PC  Can set up breakpoints, makes use of serial communication

 JTAG port (Joint Test Action Group)

25

Overview

 Debugging techniques  Debugging a distributed system  Preparation for the exam

Distributed systems

 Example: wireless sensor network deployment  Setup:

 Large scale network (tens, hundreds of devices)  Radio communication  Difficult to access, outside deployments  Distributed algorithms

27

Jan Beutel: Deployment techniques for Sensor Networks

Typical deployments

28

Typical platforms

29

Example: Great Duck Island

 Details

 Summer 2002, Great Duck Island,

gulf of Maine, USA

 5000 burrows of Leach’s Storm Petrels  Hardware

 43 nodes deployed for four months  Sensors: light, temperature, humidity, pressure, infrared  Solar powered gateway connected to a basestation (satellite

connection)

 123 days of experiment -> 1.1M samples (6.6M expected)

30

GDI experiment

 Failures:

 Hardware: water entered casings  Sensors shared A/D converter

 One malfunctioning sensor corrupted all readings on nodes

 Transparent casing: high temperature inside  Clocks & oscillators not working as expected

 Second deployment

 Multihop network

 Density too high: batteries drained

 Basestation failure due to harsh weather

31

LOFAR-Agro

 Summer 2005, Holland,

precision agriculture

 “Murphy loves potatoes”

 Wrong commit to SVN

 Nodes were flashed with buggy code  Software update delivered (and stored in external memory)

 Bug lead to continuous update of the network  Batteries dead after 4 days

 Routing and MAC layers used different buffer sizes  Watchdog protection -> triggered on all nodes within 2-6h  Data collection – 2% 

 Gateway issues: power outage in the morning (solar)  Nodes stored data also in FLASH -> bug

32

Deployment problems

 Node problems

 Low battery corrupt readings, software resets  Counter overflows, WDT, incorrect downloads, sinks

 Link problems and paths problems

 Message loss, network congestions, asymmetric links  Networking protocols, routing loops

 Global problems

 Partitioned networks, emergent behavior (resulting in latency)

33

Understanding the system

 Debugging does not involve only software!  Four components

 Hardware  Software  Communication  Environment

34

Node instrumentation

 Software instrumentation

 Extract the (partial) state of the system  Source vs. Binary

 Probe effects, trampoline

 Operating system vs. Application  Dynamic instrumentation  Aspect oriented programming

 User specifies patterns

 Hardware instrumentation

35

Network instrumentation

36

Analyzing the system

 Monitoring and visualization  Inferring network state from node state  Failure detection  Root cause analysis  Node-level debugging  Replay and checkpointing

37

Overview

 Debugging techniques  Debugging a distributed system  Preparation for the exam

Exam

 Details

 Material: Book + 2 articles + lecture notes (see website)  True/False questions  English language  Questions/clarifications if needed – let me know!

 DO register for the exam  You need to pass the lab for entering the exam (≥5)  You need to pass the exam to get a mark (≥5)  Final mark: 50% written exam; 50% practicum mark

39

Question (old school)

Which of the following statements is NOT correct? An Embedded System …

 a) has always human safety requirements  b) may have interactions with the physical world  c) may control controlling a technical entity  d) is embedded in the physical world

Question (new style)

An Embedded System may control controlling a technical entity. True/False?

Question

Using interrupts improves context switch times. True/False?

Question

Using interrupts improves response times. True/False?

Question

This code is an example of an RRI architecture. True/False?

void f1(void) { delay(1000); } void f2(void) { delay(2000); } void f3(void) { delay(3000); } void f4(void) { delay(4000); } void main (void) { while (1) { if (buttons & 0x01) f1(); if (buttons & 0x02) f2(); if (buttons & 0x04) f3(); if (buttons & 0x08) f4(); delay(1000); } }

45

Thank you for taking the TI2726-B course!