Request Tracking in DROPS (Diplomverteidigung) Bj orn D obel - - PowerPoint PPT Presentation

▶
request tracking in drops
SMART_READER_LITE
LIVE PREVIEW

Request Tracking in DROPS (Diplomverteidigung) Bj orn D obel - - PowerPoint PPT Presentation

Apr 03, 2023 326 likes •774 views

Introduction Instrumentation Applications Evaluation Conclusion and Outlook Request Tracking in DROPS (Diplomverteidigung) Bj orn D obel <doebel@os.inf.tu-dresden.de> 23.06.2006 Bj orn D obel Request tracking in DROPS

slide-1
SLIDE 1

Introduction Instrumentation Applications Evaluation Conclusion and Outlook

Request Tracking in DROPS

(Diplomverteidigung)

Bj¨

  • rn D¨
  • bel

<doebel@os.inf.tu-dresden.de> 23.06.2006

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 1 / 27

slide-2
SLIDE 2

Introduction Instrumentation Applications Evaluation Conclusion and Outlook

Contents

1

Introduction

2

Instrumentation

3

Applications

4

Evaluation

5

Conclusion and Outlook

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 2 / 27

slide-3
SLIDE 3

Introduction Instrumentation Applications Evaluation Conclusion and Outlook Motivation

Motivation

Donald Knuth:

”Premature optimization is the root of all evil (or at least most of it) in programming.”

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 3 / 27

slide-4
SLIDE 4

Introduction Instrumentation Applications Evaluation Conclusion and Outlook Motivation

Motivation (2)

Performance optimizations mostly inside the kernel

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 4 / 27

slide-5
SLIDE 5

Introduction Instrumentation Applications Evaluation Conclusion and Outlook Motivation

Motivation (2)

Performance optimizations mostly inside the kernel What about the components on top of Fiasco?

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 4 / 27

slide-6
SLIDE 6

Introduction Instrumentation Applications Evaluation Conclusion and Outlook Motivation

Motivation (2)

Performance optimizations mostly inside the kernel What about the components on top of Fiasco? Enable evaluation and verification of

(inter-)component behavior (inter-)component resource usage (inter-)component performance

Use offline monitoring for evaluation Use online monitoring for verification

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 4 / 27

slide-7
SLIDE 7

Introduction Instrumentation Applications Evaluation Conclusion and Outlook Definitions

Definitions

Event An event contains data representing an action that happens at a certain point in time.

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 5 / 27

slide-8
SLIDE 8

Introduction Instrumentation Applications Evaluation Conclusion and Outlook Definitions

Definitions

Event An event contains data representing an action that happens at a certain point in time. Sensor A sensor is a means to produce events, mostly some piece of software.

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 5 / 27

slide-9
SLIDE 9

Introduction Instrumentation Applications Evaluation Conclusion and Outlook Definitions

Definitions

Event An event contains data representing an action that happens at a certain point in time. Sensor A sensor is a means to produce events, mostly some piece of software. Request A request is a structured set of events. Requests are interesting with respect to resource usage.

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 5 / 27

slide-10
SLIDE 10

Introduction Instrumentation Applications Evaluation Conclusion and Outlook Instrumentation points

Where to instrument?

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 6 / 27

slide-11
SLIDE 11

Introduction Instrumentation Applications Evaluation Conclusion and Outlook Existing facilities

What do we have?

Fiasco tracebuffer

needed for kernel events inflexible

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 7 / 27

slide-12
SLIDE 12

Introduction Instrumentation Applications Evaluation Conclusion and Outlook Existing facilities

What do we have?

Fiasco tracebuffer

needed for kernel events inflexible

Ferret monitoring framework

sensor directory predefined sensors

scalars histograms lists

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 7 / 27

slide-13
SLIDE 13

Introduction Instrumentation Applications Evaluation Conclusion and Outlook Instrumentation rules

Instrumentation rules

1 Deterministic event layout Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 8 / 27

slide-14
SLIDE 14

Introduction Instrumentation Applications Evaluation Conclusion and Outlook Instrumentation rules

Instrumentation rules

1 Deterministic event layout 2 Use large events if possible

less calls to monitoring framework

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 8 / 27

slide-15
SLIDE 15

Introduction Instrumentation Applications Evaluation Conclusion and Outlook Instrumentation rules

Instrumentation rules

1 Deterministic event layout 2 Use large events if possible

less calls to monitoring framework

3 Keep out monitoring overhead

need to extract overhead from measurements might be necessary to obtain more data

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 8 / 27

slide-16
SLIDE 16

Introduction Instrumentation Applications Evaluation Conclusion and Outlook IPC tracing

IPC tracing

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 9 / 27

slide-17
SLIDE 17

Introduction Instrumentation Applications Evaluation Conclusion and Outlook IPC tracing

IPC tracing

Automated instrumentation of IPC using a Dice tracing plugin: dice trace

covers most L4 applications

Sensor support library Global list sensor for Dice events

initialized during application startup

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 10 / 27

slide-18
SLIDE 18

Introduction Instrumentation Applications Evaluation Conclusion and Outlook IPC tracing

IPC tracing (2)

Client-side: one event for each call

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 11 / 27

slide-19
SLIDE 19

Introduction Instrumentation Applications Evaluation Conclusion and Outlook IPC tracing

IPC tracing (2)

Client-side: one event for each call Server-side: one event for each client-side call

tricky: allow reply only

store pending events at server efficient lookup

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 11 / 27

slide-20
SLIDE 20

Introduction Instrumentation Applications Evaluation Conclusion and Outlook IPC tracing

IPC tracing (2)

Client-side: one event for each call Server-side: one event for each client-side call

tricky: allow reply only

store pending events at server efficient lookup

alternative: split server-side event into multiple ones

multiple times of monitoring overhead

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 11 / 27

slide-21
SLIDE 21

Introduction Instrumentation Applications Evaluation Conclusion and Outlook IPC tracing

IPC tracing: event layout

Timestamp Event head Major number Minor number Instance number CPU ID Producer ID Producer ID Additional data Start timestamp Start timestamp Stop timestamp Stop timestamp Send size Component start Receive size Dice opcode Component stop

  • Client-side

Server-side

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 12 / 27

slide-22
SLIDE 22

Introduction Instrumentation Applications Evaluation Conclusion and Outlook IPC tracing

L4Env tracing

L4Env services used by nearly all applications Built-in instrumentation can help with profiling and debugging Global L4Env sensor

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 13 / 27

slide-23
SLIDE 23

Introduction Instrumentation Applications Evaluation Conclusion and Outlook IPC tracing

L4Env tracing

L4Env services used by nearly all applications Built-in instrumentation can help with profiling and debugging Global L4Env sensor Example: semaphores

1

Critical sections:

How much contention? How long are critical sections?

2

Consumer–producer scenarios:

How long are consumers/producers waiting for a semaphore?

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 13 / 27

slide-24
SLIDE 24

Introduction Instrumentation Applications Evaluation Conclusion and Outlook L4Linux tracing

Tracing L4Linux

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 14 / 27

slide-25
SLIDE 25

Introduction Instrumentation Applications Evaluation Conclusion and Outlook L4Linux tracing

Tracing the L4Linux kernel

Several approaches to Linux kernel tracing:

printk()

manual instrumentation flexible

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 15 / 27

slide-26
SLIDE 26

Introduction Instrumentation Applications Evaluation Conclusion and Outlook L4Linux tracing

Tracing the L4Linux kernel

Several approaches to Linux kernel tracing:

printk()

manual instrumentation flexible

LTT / LTTng

fixed amount of predefined events visualization backend

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 15 / 27

slide-27
SLIDE 27

Introduction Instrumentation Applications Evaluation Conclusion and Outlook L4Linux tracing

Tracing the L4Linux kernel

Several approaches to Linux kernel tracing:

printk()

manual instrumentation flexible

LTT / LTTng

fixed amount of predefined events visualization backend

kProbes

instrument arbitrary locations inside the kernel by patching instructions with int3 flexible

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 15 / 27

slide-28
SLIDE 28

Introduction Instrumentation Applications Evaluation Conclusion and Outlook L4Linux tracing

Tracing the L4Linux kernel (2)

Adam Lackorzynski ported kProbes to L4Linux:

cannot use int3 as break instruction use hlt instead

Ferret sensor inside the L4Linux kernel kProbes overhead higher than in native Linux, but constant Differential analysis: run the same kProbe in L4Linux and Linux

FerretLX kernel module for Linux

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 16 / 27

slide-29
SLIDE 29

Introduction Instrumentation Applications Evaluation Conclusion and Outlook L4Linux tracing

Tracing L4Linux processes

No user-space kProbes exist yet Manual instrumentation possible:

map a Ferret sensor into task’s memory

1

direct request from sensor directory

2

map to a fixed area in task’s address space

3

use L4Linux as a proxy for Ferret sensors

2) implemented by Martin Pohlack

Need to modify source code

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 17 / 27

slide-30
SLIDE 30

Introduction Instrumentation Applications Evaluation Conclusion and Outlook L4Linux scenario

Inspecting fork/vfork in (L4)Linux

kProbes at certain locations inside do fork() routine:

call to alloc pid() call to wake up new task() function return

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 18 / 27

slide-31
SLIDE 31

Introduction Instrumentation Applications Evaluation Conclusion and Outlook L4Linux scenario

Inspecting fork/vfork in (L4)Linux

kProbes at certain locations inside do fork() routine:

call to alloc pid() call to wake up new task() function return

More probes:

detect system calls:

l4x dispatch syscall() (inline...) system call

ret from fork - start instruction of child process exit system call

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 18 / 27

slide-32
SLIDE 32

Introduction Instrumentation Applications Evaluation Conclusion and Outlook L4Linux scenario

Inspecting fork/vfork in (L4)Linux

kProbes at certain locations inside do fork() routine:

call to alloc pid() call to wake up new task() function return

More probes:

detect system calls:

l4x dispatch syscall() (inline...) system call

ret from fork - start instruction of child process exit system call

Context switches (Fiasco tracebuffer, switch to()) Defined Magpie schema for visualization Found a peculiarity in L4Linux

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 18 / 27

slide-33
SLIDE 33

Introduction Instrumentation Applications Evaluation Conclusion and Outlook L4Linux scenario Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 19 / 27

slide-34
SLIDE 34

Introduction Instrumentation Applications Evaluation Conclusion and Outlook String IPC

String IPC performance

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 20 / 27

slide-35
SLIDE 35

Introduction Instrumentation Applications Evaluation Conclusion and Outlook String IPC

String IPC performance (2)

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 21 / 27

slide-36
SLIDE 36

Introduction Instrumentation Applications Evaluation Conclusion and Outlook L4Linux

Evaluating L4Linux

UnixBench on L4Linux and native Linux:

system call overhead: 8x slower process creation: 5x slower pipe throughput: 3x slower

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 22 / 27

slide-37
SLIDE 37

Introduction Instrumentation Applications Evaluation Conclusion and Outlook L4Linux

Evaluating L4Linux

UnixBench on L4Linux and native Linux:

system call overhead: 8x slower process creation: 5x slower pipe throughput: 3x slower

System call execution time (in cycles): native Linux L4Linux sys brk 1,800 2,800 sys llseek 1,089 1,276 sys open 20,865 21,579 sys clone 62,604 161,775

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 22 / 27

slide-38
SLIDE 38

Introduction Instrumentation Applications Evaluation Conclusion and Outlook L4Linux

Evaluating L4Linux

UnixBench on L4Linux and native Linux:

system call overhead: 8x slower process creation: 5x slower pipe throughput: 3x slower

System call execution time (in cycles): native Linux L4Linux sys brk 1,800 2,800 sys llseek 1,089 1,276 sys open 20,865 21,579 sys clone 62,604 161,775 Two main reasons:

two context switches per system call interaction with L4 servers

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 22 / 27

slide-39
SLIDE 39

Introduction Instrumentation Applications Evaluation Conclusion and Outlook L4Linux

Context switch issues

Hypothesis: Context switch overhead decreases performance. 100,000 executions of sys getpid()

native Linux L4Linux User-space execution 274 3,778 time in cycles (avg) Instruction cache misses 544 1,703,332 for 100,000 calls Data Cache misses 286 402,398 for 100,000 calls ITLB misses for 100,000 calls 35 501,679 DTLB misses for 100,000 calls 141 1,052,520

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 23 / 27

slide-40
SLIDE 40

Introduction Instrumentation Applications Evaluation Conclusion and Outlook L4Linux

L4Linux task creation

Hypothesis: Cooperation with L4 servers can be optimized. implemented transfer of chief rights to L4Linux server implemented a task cache in L4Linux

L4Linux + new task + task caching unmodified management fork() 330,425 292,857 255,087 vfork() 45,502 44,274 45,510 execve() 237,014 243,574 207,376

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 24 / 27

slide-41
SLIDE 41

Introduction Instrumentation Applications Evaluation Conclusion and Outlook Conclusion

Conclusion

Request tracking throughout DROPS components Applicable for online and offline monitoring Useful for performance evaluation

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 25 / 27

slide-42
SLIDE 42

Introduction Instrumentation Applications Evaluation Conclusion and Outlook Future work

Future work

LTTng for L4Linux Autonomic Computing Model-carrying code Regression performance evaluation Inspect larger scenarios

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 26 / 27

slide-43
SLIDE 43

Introduction Instrumentation Applications Evaluation Conclusion and Outlook Questions

Thank you for attending. Any Questions?

Bj¨

  • rn D¨
  • bel

Request tracking in DROPS 23.06.2006 27 / 27