Tests for Algorithmic Complexity in Programming Exercises - - PowerPoint PPT Presentation

▶
tests for algorithmic complexity in programming exercises
SMART_READER_LITE
LIVE PREVIEW

Tests for Algorithmic Complexity in Programming Exercises - - PowerPoint PPT Presentation

Nov 03, 2023 641 likes •900 views

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Tests for Algorithmic Complexity in Programming Exercises Intermediate talk for the Guided Research by Stefan Wlfert advised by Johannes

slide-1
SLIDE 1

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Tests for Algorithmic Complexity in Programming Exercises

Intermediate talk for the Guided Research by

Stefan Wölfert

advised by Johannes Naab and Henning Stubbe Friday 17th January, 2020 Chair of Network Architectures and Services Department of Informatics Technical University of Munich

slide-2
SLIDE 2

Tests for Algorithmic Complexity in Programming Exercises

Programming exercises in GRNVS:

  • Automatic testing of student submissions for functional requirements
  • Non-functional assessment by hand
  • S. Wölfert — ethstats runtime test case

2

slide-3
SLIDE 3

Tests for Algorithmic Complexity in Programming Exercises

Programming exercises in GRNVS:

  • Automatic testing of student submissions for functional requirements
  • Non-functional assessment by hand

Problem statement:

  • Finding a method for estimating the used data structure
  • Creating a new test case for ethstats programming exercise
  • S. Wölfert — ethstats runtime test case

2

slide-4
SLIDE 4

Programming exercise: ethstats

Problem statement

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 0 B 4 B 8 B 12 B Destination Address Source Address Ethertype

Figure 1: IEEE 802.3 Ethernet Header 1

1GRNVS Cheatsheet

  • S. Wölfert — ethstats runtime test case

3

slide-5
SLIDE 5

Programming exercise: ethstats

Problem statement

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 0 B 4 B 8 B 12 B Destination Address Source Address Ethertype

Figure 1: IEEE 802.3 Ethernet Header 1

Tasks:

  • Reading data from a raw socket
  • Evaluating the destination address
  • Determining the EtherType
  • Counting amount of frames and data

per EtherType Sample output:

I am ready! 0x0800: 3 frames, 192 bytes 0x0806: 3 frames, 209 bytes 0x86dd: 4 frames, 256 bytes 3 of them were for me 4 of them were multicast IPv4 accounted for 29.2% and IPv6 for 39.0% of traffic

1GRNVS Cheatsheet

  • S. Wölfert — ethstats runtime test case

3

slide-6
SLIDE 6

Programming exercise: ethstats

Test cases Current test cases:

  • Handling EtherTypes
  • Output format
  • Correct counting
  • Correct destination evaluation
  • Appropriate runtime
  • S. Wölfert — ethstats runtime test case

4

slide-7
SLIDE 7

Programming exercise: ethstats

Test cases Current test cases:

  • Handling EtherTypes
  • Output format
  • Correct counting
  • Correct destination evaluation
  • Appropriate runtime

Additional requirements: choosing an appropriate data structure

  • S. Wölfert — ethstats runtime test case

4

slide-8
SLIDE 8

Scaling of data structures

Runtime behaviour Time-determining operations:

  • Data structure independent input and output behaviour
  • Main influencing operations: insert, update (s. Table 1 and Table 2)

Table 1: Best-case runtime behaviour for a single operation.

Map Linked-List Insert O(1) O(1) Update O(1) O(1)

Table 2: Worst-case runtime behaviour for a single operation.

Map Linked-List Insert O(1) O(n) Update O(1) O(n)

  • S. Wölfert — ethstats runtime test case

5

slide-9
SLIDE 9

Scaling of data structures

Scaling behaviour

2 4 6 8 10 2 4 6 8 10

O(n2) O(n)

n scaling

Figure 2: Sample scaling behaviour.

  • S. Wölfert — ethstats runtime test case

6

slide-10
SLIDE 10

Runtime test case

Goal: estimating the used data structure by runtime measurements

  • Reference implementations for ethstats in c and java with array and linked-list data struc-

ture

  • Creating a testbed for runtime measurements
  • Using linear scaling behaviour as main criterion for estimation
  • S. Wölfert — ethstats runtime test case

7

slide-11
SLIDE 11

Runtime test case

Testbed eth1 eth1 eth0 eth0

Figure 3: Network topology for tests on ethstats

  • S. Wölfert — ethstats runtime test case

8

slide-12
SLIDE 12

Runtime test case

Testbed eth1 eth1 eth0 eth0

Figure 3: Network topology for tests on ethstats

Workflow:

  • 1. Providing clean environment on testee
  • 2. Building ethstats on testee
  • 3. Sending ready-marker on control channel
  • 4. Sending frames on data channel in ascending and descending order, while observing user

space runtime via /usr/bin/time

  • 5. Evaluating output and measured time values
  • S. Wölfert — ethstats runtime test case

8

slide-13
SLIDE 13

Runtime test case

Parameters Influences on the measurement:

  • Frame size
  • Sending rate and time-out value
  • Amount of frames
  • S. Wölfert — ethstats runtime test case

9

slide-14
SLIDE 14

Runtime test case

Parameters Influences on the measurement:

  • Frame size
  • Sending rate and time-out value
  • Amount of frames

2 4 6 8 10 12 0.2 0.4 0.6 0.8 1 amount of frames [2n]

  • avg. user space time [s]

c map c linked-list java map java linked-list Figure 4: Runtime behaviour of reference implementations.

  • S. Wölfert — ethstats runtime test case

9

slide-15
SLIDE 15

Runtime test case

Simple linear regression

  • For a single point:

ti = β · xi + t0 + ǫi (1)

2 4 6 8 10 2 4 6 8 10

O(n2) O(n)

n scaling

  • S. Wölfert — ethstats runtime test case

10

slide-16
SLIDE 16

Runtime test case

Simple linear regression

  • For a single point:

ti = β · xi + t0 + ǫi (1)

  • Estimating β and t0 by method of least squares:

β = n

i=1

  • ti − ¯

t n

i=1 (xi − ¯

x)2 t0 = ¯ t − β · ¯ x (2)

2 4 6 8 10 2 4 6 8 10

O(n2) O(n)

n scaling

  • S. Wölfert — ethstats runtime test case

10

slide-17
SLIDE 17

Runtime test case

Simple linear regression

  • For a single point:

ti = β · xi + t0 + ǫi (1)

  • Estimating β and t0 by method of least squares:

β = n

i=1

  • ti − ¯

t n

i=1 (xi − ¯

x)2 t0 = ¯ t − β · ¯ x (2)

  • Spreading due to regression SQR:

SQR =

n

  • i=1
  • ˜

ti − ¯ t2 ˜ ti = ti − ǫi (3)

2 4 6 8 10 2 4 6 8 10

O(n2) O(n)

n scaling

  • S. Wölfert — ethstats runtime test case

10

slide-18
SLIDE 18

Runtime test case

Simple linear regression

  • For a single point:

ti = β · xi + t0 + ǫi (1)

  • Estimating β and t0 by method of least squares:

β = n

i=1

  • ti − ¯

t n

i=1 (xi − ¯

x)2 t0 = ¯ t − β · ¯ x (2)

  • Spreading due to regression SQR:

SQR =

n

  • i=1
  • ˜

ti − ¯ t2 ˜ ti = ti − ǫi (3)

  • Total spreading SQT:

SQT =

n

  • i=1
  • ti − ¯

t2 (4)

2 4 6 8 10 2 4 6 8 10

O(n2) O(n)

n scaling

  • S. Wölfert — ethstats runtime test case

10

slide-19
SLIDE 19

Runtime test case

Simple linear regression

  • For a single point:

ti = β · xi + t0 + ǫi (1)

  • Estimating β and t0 by method of least squares:

β = n

i=1

  • ti − ¯

t n

i=1 (xi − ¯

x)2 t0 = ¯ t − β · ¯ x (2)

  • Spreading due to regression SQR:

SQR =

n

  • i=1
  • ˜

ti − ¯ t2 ˜ ti = ti − ǫi (3)

  • Total spreading SQT:

SQT =

n

  • i=1
  • ti − ¯

t2 (4)

  • Quality of fitting η:

η = 1 − SQR SQT (5)

2 4 6 8 10 2 4 6 8 10

O(n2) O(n)

n scaling

  • S. Wölfert — ethstats runtime test case

10

slide-20
SLIDE 20

Runtime test case

Testing reference implementations User space time measurement:

  • Tested by sending 512, 1024, 2048, 3072, 4096 and 5120 frames
  • 5 Repetitions

Table 3: Fit quality of reference implementations.

language data rate(asc) rate(dsc) tmax structure [%] [%] [s] c map 41.6 25.6 0.05 c list 95.7 32.9 0.19 java map 90.1 89.6 0.83 java list time-out time-out time-out

  • S. Wölfert — ethstats runtime test case

11

slide-21
SLIDE 21

Runtime test case

Testing reference implementations User space time measurement:

  • Tested by sending 512, 1024, 2048, 3072, 4096 and 5120 frames
  • 5 Repetitions

Table 3: Fit quality of reference implementations.

language data rate(asc) rate(dsc) tmax structure [%] [%] [s] c map 41.6 25.6 0.05 c list 95.7 32.9 0.19 java map 90.1 89.6 0.83 java list time-out time-out time-out Test criteria:

  • Small difference in quality rate for appropriate implementations
  • Small maximum runtime as second alternative criterion
  • S. Wölfert — ethstats runtime test case

11

slide-22
SLIDE 22

Runtime test case

Testing 2019SS students submissions User space time measurement:

  • Tested by sending 512, 1024, 2048, 3072, 4096 and 5120 frames
  • 5 Repetitions

Table 4: Identifying students submissions based on simple linear regression and total time in user space.

language data false correct structure identification identification c map 5 c list 8 1 java map 2 43 java list 9 2

  • S. Wölfert — ethstats runtime test case

12

slide-23
SLIDE 23

Runtime test case

Testing 2019SS students submissions User space time measurement:

  • Tested by sending 512, 1024, 2048, 3072, 4096 and 5120 frames
  • 5 Repetitions

Table 4: Identifying students submissions based on simple linear regression and total time in user space.

language data false correct structure identification identification c map 5 c list 8 1 java map 2 43 java list 9 2

  • Evaluation of c and java implementations only
  • Multiple crashes of test framework causing missing data
  • S. Wölfert — ethstats runtime test case

12

slide-24
SLIDE 24

Summary

  • Runtime measurements possible for evaluating chosen data structures in ethstats pro-

gramming exercise

  • Offering expected O(n) runtime behaviour for map based reference implementations
  • Simple linear regression as main-criterion providing implementation independence
  • Robuster testing framework needed for students submissions
  • Further tests with implementations in other programming languages than c or java needed

in order to set final test criteria

  • S. Wölfert — ethstats runtime test case

13