Zero-Copy Socket Splicing Alexander Bluhm bluhm@openbsd.org - - PowerPoint PPT Presentation

▶
zero copy socket splicing
SMART_READER_LITE
LIVE PREVIEW

Zero-Copy Socket Splicing Alexander Bluhm bluhm@openbsd.org - - PowerPoint PPT Presentation

Dec 28, 2022 344 likes •737 views

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications Zero-Copy Socket Splicing Alexander Bluhm bluhm@openbsd.org Sunday, 29. September 2013 Motivation Kernel MBuf Packet Processing Socket

slide-1
SLIDE 1

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Zero-Copy Socket Splicing

Alexander Bluhm

bluhm@openbsd.org

Sunday, 29. September 2013

slide-2
SLIDE 2

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Agenda

1

Motivation

2

Kernel MBuf

3

Packet Processing

4

Socket Splicing

5

Interface

6

Implementation

7

Applications

slide-3
SLIDE 3

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Agenda

1

Motivation

2

Kernel MBuf

3

Packet Processing

4

Socket Splicing

5

Interface

6

Implementation

7

Applications

slide-4
SLIDE 4

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Application Level Gateway

Physical Data Link Network IP TCP/UDP Application Packet Filter Relay Kernel User Land Socket Splicing

slide-5
SLIDE 5

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Persistent HTTP Filtering

Body

content length

Header Body

content length

Header

copy copy copy filter copy copy filter

slide-6
SLIDE 6

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

HTTP Socket Splicing

Body

splice length

Header Body

splice length

Header

splice filter splice filter

Kernel User Land

slide-7
SLIDE 7

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Agenda

1

Motivation

2

Kernel MBuf

3

Packet Processing

4

Socket Splicing

5

Interface

6

Implementation

7

Applications

slide-8
SLIDE 8

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

MBuf Data

mbuf m hdr m data m len m dat ether header ip header udp header size 256 42 size 236 size 42

slide-9
SLIDE 9

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

MBuf Data Chaining

mbuf m hdr m next m data m len m pkthdr len m pktdat ether header ip header udp header size 256 42 142 size 196 size 42 mbuf m hdr m next m data m len m dat payload size 256 NULL 100 size 236 size 100

slide-10
SLIDE 10

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

MBuf Packet Chaining

mbuf m hdr m next m nextpkt m pkthdr mbuf m hdr m next m nextpkt mbuf m hdr m next m nextpkt mbuf m hdr m next m nextpkt m pkthdr mbuf m hdr m next m nextpkt

slide-11
SLIDE 11

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

MBuf Cluster

mbuf m hdr m data m len m pkthdr m ext ext buf ext size size 256 1400 2048 ether header ip header udp header payload size 2048 size 1400

slide-12
SLIDE 12

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

MBuf Cluster Copy

mbuf m data ext buf mbuf m data ext buf ether header ip header udp header payload

slide-13
SLIDE 13

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Agenda

1

Motivation

2

Kernel MBuf

3

Packet Processing

4

Socket Splicing

5

Interface

6

Implementation

7

Applications

slide-14
SLIDE 14

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Packet Input

network driver interrupt handler ether input() ip interface receive queue, m nextpkt ip input() inetsw[] internet protocol switch tcp input() socket receive buffer, m next soreceive() read() Kernel User Land

slide-15
SLIDE 15

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Packet Output

write() sosend() socket send buffer, m next tcp output() ip output() ether output() interface send queue, m nextpkt if start() network driver start routine Kernel User Land

slide-16
SLIDE 16

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Data Copy

tcp input() so rcv soreceive() uiomove() copyout() read() write() copyin() uiomove() sosend() so snd tcp output() Relay

slide-17
SLIDE 17

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Process Wakeup

tcp input() so rcv soreceive() struct socket file descriptor read() select() write() sosend() so snd tcp output() sorwakeup() sowwakeup() ACK

slide-18
SLIDE 18

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Agenda

1

Motivation

2

Kernel MBuf

3

Packet Processing

4

Socket Splicing

5

Interface

6

Implementation

7

Applications

slide-19
SLIDE 19

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Socket Splicing

tcp input() so rcv somove() sosplice() setsockopt(SO SPLICE) so snd tcp output() tcp input() sorwakeup() sowwakeup() ACK

slide-20
SLIDE 20

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

UDP Sockets

udp input() so rcv soreceive() somove() sosend() udp output()

slide-21
SLIDE 21

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Layer

ipintrq ip input() tcp input() so rcv soreceive() read() write() sosend() so snd tcp output() ip output() if snd Relaying Forwarding Socket Splicing

slide-22
SLIDE 22

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Agenda

1

Motivation

2

Kernel MBuf

3

Packet Processing

4

Socket Splicing

5

Interface

6

Implementation

7

Applications

slide-23
SLIDE 23

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Simple API

Begin splicing from source to drain setsockopt(source fd, SO SPLICE, drain fd) Stop splicing setsockopt(source fd, SO SPLICE, -1) Get spliced data length getsockopt(source fd, SO SPLICE, &length)

slide-24
SLIDE 24

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Extended API

struct splice { int sp_fd; /* drain */

  • ff_t

sp_max; /* maximum */ struct timeval sp_idle; /* timeout */ }; setsockopt(source fd, SO SPLICE, &splice)

slide-25
SLIDE 25

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Properties

Splicing is unidirectional Invoke it twice for bidirectional splicing Process can turn it on and off Works for TCP and UDP Can mix IPv4 and IPv6 sockets

slide-26
SLIDE 26

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Unsplice

Dissolve socket splicing manually read(2) or select(2) from the source EOF source socket shutdown EPIPE drain socket error EFBIG maximum data length ETIMEDOUT idle timeout

slide-27
SLIDE 27

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Agenda

1

Motivation

2

Kernel MBuf

3

Packet Processing

4

Socket Splicing

5

Interface

6

Implementation

7

Applications

slide-28
SLIDE 28

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Struct Socket

struct socket { ... struct socket *so_splice; struct socket *so_spliceback;

  • ff_t

so_splicelen;

  • ff_t

so_splicemax; struct timeval so_idletv; struct timeout so_idleto; ... };

slide-29
SLIDE 29

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

sosplice(9)

Protocol must match Sockets must be connected Double link sockets Move existing data

slide-30
SLIDE 30

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

somove(9)

Check for errors Check for space Handle maximum Handle out of band data Move socket buffer data

slide-31
SLIDE 31

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

sounsplice()

Manual unsplice Cannot receive Cannot send Maximum Timeout Socket closed

slide-32
SLIDE 32

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

sorwakeup() sowwakeup()

Called from tcp input() Source calls sorwakeup() Drain calls sowwakeup() Both invoke somove(9)

slide-33
SLIDE 33

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Agenda

1

Motivation

2

Kernel MBuf

3

Packet Processing

4

Socket Splicing

5

Interface

6

Implementation

7

Applications

slide-34
SLIDE 34

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Relayd

Plain TCP connections HTTP connections Filter persistent HTTP HTTP Chunking

slide-35
SLIDE 35

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Tests

/usr/src/regress/sys/kern/sosplice/ 15 API tests 18 UDP tests 76 TCP tests perf/relay.c simple example BSD::Socket::Splice Perl API 28 relayd tests

slide-36
SLIDE 36

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Performance

Factor 1 or 2 for TCP Factor 6 or 8 for UDP

slide-37
SLIDE 37

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Documentation

Manpage setsockopt(2) SO SPLICE Manpage sosplice(9) somove(9)

slide-38
SLIDE 38

Motivation Kernel MBuf Packet Processing Socket Splicing Interface Implementation Applications

Questions

?