Real-Time Systems: Testing Your System in Linux Jiangnan Liu CSE - - PowerPoint PPT Presentation

Real-Time Systems: Testing Your System in Linux

Jiangnan Liu CSE 520S Spring 02/13/2020

With adaptions from Haoran Li

What is meant by Time?

Ø Absolute time since some fixed past event, e.g.:

q Seconds since start of the Unix Epoch

(00:00:00 UTC on 1 January 1970)

q Seconds since system boot

Ø Relative time, E.g.:

q Seconds between two events q Ten seconds into the future (from now) q Execution time of a program segment

Ø World time, E.g.:

q January 30th, 9:00 AM

Ø An OS must approximate time to provide time-based functions for users.

2

Measure Elapsed Time

Why Elapsed Time First?

Ø End-to-End Latency

q Get absolute time on each end?

Ø Elapsed Time:

q Round Trip Time (ping) q Same Clock Source

4

Clock 1 Clock 2 Start_time (clock 1) end_time (clock 2) Synchronized / Asynchronized Clocks? Start_time (clock 1) End_time (clock 1) Clock 1 Measure Time on a single Host

Get a coarse-grained estimation

Ø Use shell built-in command

q time q “real”: Wall Time Elapsed (Just an estimation, Don’t rely on it) q “user”: Execution Time in User Space q “sys”: Execution Time in Kernel Space (syscall)

5

Get an overview of your program’s response / execution time.

Measure Elapsed Time: gettimeofday()

Ø gettimeofday()

[http://linux.die.net/man/2/sched_setscheduler]. Wall clock time

q return struct timeval, includes tv_sec and tv_usec q NOT ok for measuring overhead on standard kernel configuration q Wall clock time is subject to anomalies

• User/other program(NTP) changes clock

6

Normal Evaluation (5 seconds) Intentionally change the wall clock, when program is running

Don’t Relied on Wall Clock Time

Ø Shell built-in command “time”

7

When Measuring Elapse Time, don’t rely on WCT.

Use POSIX clock_gettime()

Ø Sources: https://linux.die.net/man/3/clock_gettime

q CLOCK_REALTIME:

• Wall Time, affected by discontinuous jump

q CLOCK_MONOTONIC:

• Not affected by discontinuous jump but affected by

incremental adjustments (e.g. NTP).

• Clock cannot jump, but may skew

q CLOCK_MONOTONIC_RAW:

• Not affected by NTP
• More accurate for very short intervals

Ø Precision: clock_getres()

8

For short latency measurement in a single host, try CLOCK_MONOTIC_RAW

FYI: Other Time Sources

RTC (Real-Time Clock)

q Available on most computers (not on RPi 2 or 3 unless you add it) q Low precision (as low as 0.5 seconds)

Hardware Timers

q Might be used to generate interrupts, might be queryable q Run at a variety of frequencies q Programmable Interval Timer (PIT) q High-Performance Event Timer (HPET) q Programmable Interrupt Controller (PIC) q Advanced Programmable Interrupt Controller (APIC)

Processor Cycles

q Timestamp Counter (TSC) on x86, 64-bit q Cycle Counter (CCNT) on ARM, 32-bit, 64-cycle divider, not accessible in user

mode

q Potentially very high accuracy

9

Source: CSE 522S http://www.cse.wustl.edu/~cdgill/courses/cse522/slides/03_Timers_Timing.pptx

Pointers

Ø Time related system calls in the Linux kernel

q https://0xax.gitbooks.io/linux-insides/content/Timers/timers-7.html

Ø Clock_gettime()

q Man 2 clock_gettime q https://linux.die.net/man/2/clock_gettime

Ø RDTSC

q http://www.mcs.anl.gov/~kazutomo/rdtsc.html

Ø RDTSCP

q https://www.intel.com/content/www/us/en/embedded/training/ia-32-ia-64-

benchmark-code-execution-paper.html

Ø Invariant TSC

q Pitfall of TSC usage q http://oliveryang.net/2015/09/pitfalls-of-TSC-usage/

Ø (ARM) High Resolution Timing on Raspberry Pi

q https://blog.regehr.org/archives/794

10

Measure End-to-End Latency

Why End-to-End Latency?

Ø End-to-End Latency Ø Contributor:

q propagation delay: static q queuing delays q node processing delays q routing changes

12

Clock 1 Clock 2 start_time (clock 1) end_time (clock 2)

Need synchronization between clocks

How to Get Clock Synchronized?

Ø NTP(Network Time Protocol)

q Query NTP Server, get clock synchronized

13

Clock 1 Clock 2 start_time (clock 1) end_time (clock 2) NTP Server

Background: NTP

Ø Hierarchical NTP servers: Clock Strata Ø Stratum0: reference clock Ø Stratum1: primary time servers

14

Source: https://en.wikipedia.org/wiki/Network_Time_Protocol The U.S. Naval Observatory Alternate Master Clock Stratum 0: high-precision timekeeping devices atomic (cesium) clocks

Problems with NTP

Ø Loose granularity:

q T

ens of milliseconds in public internet

q Around one millisecond in Local Area Network[1]

Ø Leap second handling:

q Detects the announcement at least a few seconds after the refclock

has started to announce the leap second

Ø Security:

q NTP has been used in Distributed Denial of Service attacks(DDos) [2]

15

Set up an NTP server: https://ubuntuforums.org/showthread.php?t=862620 https://arstechnica.com/information-technology/2014/01/new-dos-attacks-taking-down-game-sites-deliver-crippling-100-gbps-floods/

Precision Time Protocol: Better Accuracy

Ø On LAN:

q sub-microsecond range q making it suitable for measurement and control systems q PTP v.s. NTP

• hardware support present in various network interface controllers

(NIC) and network switches.

16

LAN

PTP master PTP slave PTP slave

PTP v.s. NTP

Ø NTP

q Hierarchical NTP servers q Redundancy: Client-Server, Peer-to-Peer, Multicast q Granularity:T

ens of microseconds

q Built-in in EC2

Ø PTP

q Grand Master-Slave(Master)-Slave q Redundancy: Fail take over q Hardware slaves (e.g. PCIe card) when possible q Granularity: Sub-microseconds (when w/ hardware support) q Not built-in in EC2, normally used for LAN

17

Pointers

Ø Linux PTP

q http://linuxptp.sourceforge.net/ q SO_TIMESTAMPS

q Supports the Linux PTP Hardware Clock (PHC) subsystem by using

the clock_gettime family of calls, including the new clock_adjtimex system call. clo Ø PTP daemon man page

q sudo apt-get install ptpd q man ptpd

Ø NTP v.s PTP: How do you get accuracy

q http://www.atis.org/tam/presentations/ntp_vs_ptp_darnold.pdf q https://blog.meinbergglobal.com/2013/11/22/ntp-vs-ptp-network-timing-smackdown/ q https://www.redhat.com/en/blog/combining-ptp-ntp-get-best-both-worlds

18

Summary

Ø Elapsed time or end-to-end latency? Ø What time is measured by the clock? (real, user, system, or, hopefully not, wall-clock?) Ø What is the granularity requirement of time? (s, ms, µs, ns) Ø Is the clock monotonic, or will it change with changes in the system time (via NTP, time zone, daylight savings time, by the user, etc.)? Ø Does the program interference the time? Ø Does your observation of the time interference the program?

19

Trace your System

Trace Your System by using ftrace

Ø ftrace

q Traces the internal operations of the kernel q Static tracepoints within the kernel (event tracing)

• Scheduling
• interrupts

Ø Trace-cmd

q Front-End (user-level) utility for ftrace q Example:

• sudo trace-cmd record -e sched_switch ./myapp
• Dump trace.dat

Ø Kernel Shark

q GUI trace-cmd reader

• kernelshark trace.dat

21

Kernel Shark: http://rostedt.homelinux.com/kernelshark/

A Typical Trace

22

Configure and Trace your Network

Ø ifconfig

q Configure, manage and query network interface parameters through

command line

Ø netstat

q List network(socket) connections connections to the system

Ø tcpdump

q Packet analyzer to trace packets transmitted over the network to

which the computer is attached

q Example:

• tcpdump -i eth0 dst 172.168.0.118 -w 001.pcap

Ø wireshark

q GUI based alternative to tcpdump 23

Real-Time Scheduler in Linux

Scheduling in Classes

Multiple schedulers are implemented as different scheduling classes. Normal:

q SCHED_NORMAL/SCHED_OTHER: regular, interactive CFS tasks q SCHED_BATCH: low priority, non-interactive CFS tasks q SCHED_IDLE: very low priority tasks

Real-time:

q SCHED_RR: round-robin q SCHED_FIFO: first-in, first-out q SCHED_DEADLINE: earliest deadline first

CSE 522S – Advanced Operating Systems

25

Real-Time Scheduling

26

Real-time tasks execute repeatedly (usually are periodic) under some time constraint

Task Task Task

E.g., a task is released to execute every 5 ms, and each invocation has a deadline of 5 ms Separate priority range from nice:

• Priorities range from 1 (low) to 99 (high)

0ms 5ms 10ms Time

Source: CSE 522S – Advanced Operating Systems: http://www.cse.wustl.edu/~cdgill/courses/cse522/slides/09_real_time_sched.pptx

Real-Time OS Support

Goal is to achieve predictable execution: Sources of uncertainty (and solutions):

q Scheduling preemptions (real-time scheduling) q Interrupts (can mask interrupts) q Migrations (can pin tasks to cores) q OS latency & jitter (RT_PREEMPT patch set)

Source: CSE 522S – Advanced Operating Systems: http://www.cse.wustl.edu/~cdgill/courses/cse522/slides/09_real_time_sched.pptx

27

Ideal: Real world:

Preempt Migrate

SCHED_RR

Round-robin scheduling Among tasks of equal priority: Ø Rotate through all tasks Ø Each task gets a fixed time slice Cannot run if higher priority tasks are runnable

28

Task 1 Task 2 Task 3 Time Task 1 Task 2 Task 3 Task 1 Task 2 Task 3

Source: CSE 522S – Advanced Operating Systems: http://www.cse.wustl.edu/~cdgill/courses/cse522/slides/09_real_time_sched.pptx

SCHED_FIFO

First-in, First-out scheduling Ø The first enqued task of highest priority executes to completion Ø A task will only relinquish a processor when it completes, yields, or blocks

29

Task 1 Task 2 Task 3 Time

Source: CSE 522S – Advanced Operating Systems: http://www.cse.wustl.edu/~cdgill/courses/cse522/slides/09_real_time_sched.pptx

SCHED_DEADLINE

Earliest Deadline First (EDF) scheduling Ø Whichever task has next deadline gets to run Ø Theory exists to analyze such systems Ø Linux implements bandwidth reservation to prevent deadline abuse

30

Task 3 Deadline: 8 Exec time: 2 Task 2 Deadline: 12 Exec time: 3 Task 1 Deadline: 5 Exec time: 4 Time 0 Task 1 Task 2 Task 3 5 8 12

Source: CSE 522S – Advanced Operating Systems: http://www.cse.wustl.edu/~cdgill/courses/cse522/slides/09_real_time_sched.pptx

Time 0 Task 1 Task 2 Task 3 5 8 12 Miss Deadline When using SCHED_FIFO

Scheduler Setup – Basic

Ø Two classes, would always schedule RT class first

q RT class: static priority, 1 (lowest) to 99 (highest)

• Preemptive scheduling
• SCHED_DEADLINE: Exists in Kernel, however, no libc wrapper:

§ You need write a SyscallWrapper by yourself § Or: https://github.com/jlelli/schedtool-dl

• Default Reserve 5% for other classes (50ms every 1s)

§ /proc/sys/kernel/sched_rt_period_us

1000000

§ /proc/sys/kernel/sched_rt_runtime_us

950000

q Non-RT class: SCHED_OTHER with Complete Fair Scheduler

31

Scheduler Setup – Priorities

Ø chrt command (can also check task priorities)

http://www.cyberciti.biz/faq/howto-set-real-time-scheduling-priority-process/

q sudo chrt –f –p 99 4800 # pid 4800 with priority 99 and fifo

Ø sched_scheduler [http://linux.die.net/man/2/sched_setscheduler]

32

#include <sched.h> int main() { … struct sched_param sched; sched.sched_priority = 98; if (sched_setscheduler(getpid(), SCHED_FIFO, &sched) < 0) { exit(EXIT_FAILURE); } … }

Scheduler Setup – Affinities

Ø taskset command (can also check task affinities)

[http://linux.die.net/man/2/sched_setscheduler]

q sudo taskset -c 2,3 4800 # pid 4800 runs on cores 2-3

Ø sched_setaffinity

[http://linux.die.net/man/2/sched_setscheduler]

33

#include <sched.h> int main() { … unsigned long mask = 1; if (sched_setaffinity(getpid(), sizeof(mask), &mask) < 0) { exit(EXIT_FAILURE); } … }

Pointers

Ø Linux schedulers

q https://www.kernel.org/doc/Documentation/scheduler/ q http://oreilly.com/catalog/linuxkernel/chapter/ch10.html

Ø Set priority

q chrt: http://www.cyberciti.biz/faq/howto-set-real-time-scheduling-priority-process/ q sched_scheduler: http://linux.die.net/man/2/sched_setscheduler

Ø Set CPU affinity on multi-core:

q taskset: http://linux.die.net/man/1/taskset q Check freq: http://www.advenage.com/topics/linux-timer-interrupt-frequency.php

Ø Linux real-time patches:

q Linux Preempt RT: https://rt.wiki.kernel.org/index.php/Main_Page q LITMUS-RT: http://www.litmus-rt.org/ q RTAI: https://www.rtai.org/ q SCHED_DEADLINE: http://gitorious.org/sched_deadline

34