LIO and the TCMU Userspace Passthrough: The Best of Both Worlds - - PowerPoint PPT Presentation

▶
lio and the tcmu userspace passthrough the best of both
SMART_READER_LITE
LIVE PREVIEW

LIO and the TCMU Userspace Passthrough: The Best of Both Worlds - - PowerPoint PPT Presentation

Feb 23, 2024 328 likes •616 views

LIO and the TCMU Userspace Passthrough: The Best of Both Worlds Andy Grover <agrover@redhat.com> @iamagrover March 11, 2015 1 LIO AND TCMU What is LIO? Multi-protocol in-kernel SCSI target 2 LIO AND TCMU Multi-protocol in - kernel

slide-1
SLIDE 1

LIO AND TCMU 1

LIO and the TCMU Userspace Passthrough: The Best of Both Worlds

Andy Grover <agrover@redhat.com> @iamagrover March 11, 2015

slide-2
SLIDE 2

LIO AND TCMU 2

What is LIO?

Multi-protocol in-kernel SCSI target

slide-3
SLIDE 3

LIO AND TCMU 3

Multi-protocol in-kernel SCSI target Unlike other targets like IET, tgt, and SCST, LIO is entirely kernel code.

slide-4
SLIDE 4

LIO AND TCMU 4

iSCSI commands

User Kernel disk.img /dev/vg0/vol0 Configuration Fabrics

tcm_fc iscsi

/dev/sdc

block file pscsi rtslib targetcli

LIO core

/sys/kernel/config/target

Backstores

slide-5
SLIDE 5

LIO AND TCMU 5

Why add userspace command handling?

  • Enable wider variety of backstores without

kernel code

  • Clustered network storage fjlesystems like

Ceph, GlusterFS, & other things that have shared libraries available

  • File formats beyond .img, such as qcow2 &

vmdk for more interesting features & compatibility

  • SCSI devices beyond mass storage
  • Enable experimentation just like FUSE did for

fjlesystems

slide-6
SLIDE 6

LIO AND TCMU 6

Userspace handling challenges: perf

  • I/O latency
  • I/O throughput
  • Parallelism within a dev (OoO cmd completion)
  • Parallelism across devs, we're good.
slide-7
SLIDE 7

LIO AND TCMU 7

Userspace handling challenges: usability

  • Confjguration as simple as existing backstores
  • Userspace daemon failure
  • Userspace daemon activation/restart
  • Balance ultimate fmexibility with common use
  • Avoid “multiple personalities”
  • Reasonable resource usage
slide-8
SLIDE 8

LIO AND TCMU 8

iSCSI commands

User Kernel disk.img /dev/vg0/vol0 Configuration Fabrics

tcm_fc iscsi

/dev/sdc

block file pscsi rtslib targetcli

LIO core

/sys/kernel/config/target

tcm-user

uio0 uio1

cmd handling daemon

SCSI Command processing Backstores

N E W ! N E W !

slide-9
SLIDE 9

LIO AND TCMU 9

tcm-user cmd handling daemon

/dev/uio0 /sys/class/uio/uio0 uio0

  • read to get

configuration

  • read (wait for cmd)
  • write (cmds done)
  • mmap (get SMR)

from LIO core Shared Memory Region Layout

(not to scale)

Mailbox Mailbox Command Ring Command Ring Data Area Data Area

cmdr_off cmdr_size cmd_head cmd_tail cmd_entry ...

  • pcode

iovec[] in/out data

User/Kernel Communication

... dev add/remove (netlink)

slide-10
SLIDE 10

LIO AND TCMU 10

tcm-user merged in 3.18

  • Initial patch as simple as possible
  • Performance tuning not done, BUT an interface

fmexible enough to not block likely perf

  • ptimization strategies
  • Acceptance enabled phase 2: userspace

usability pieces

slide-11
SLIDE 11

LIO AND TCMU 11

Performance Opportunities for Later

  • Larger vmalloc()ed shared mem region
  • -> Demand-allocate pages to avoid bloat
  • Block size == PAGE_SIZE preferred
  • Complete commands out of order
  • Must handle data area fragmentation
  • Fabrics copy into already-mapped pages
  • Just moves the cache misses?
  • Fabrics don't have per-lun device queues
  • anyway. Just IB reimplemented poorly?
  • Userspace busywait on ring cmd_head
slide-12
SLIDE 12

LIO AND TCMU 12

Our user-kernel API ended up fmexible, but fraught with danger!

  • Ring operations easy to mess up
  • Make every handler write daemon boilerplate?
  • And support Netlink?
  • And maybe D-Bus???
slide-13
SLIDE 13

LIO AND TCMU 13

tcmu-runner: A standard handler daemon

  • Handle the messy ring bits
  • Expose a C plugin API
  • Implement library routines for common handler

code, e.g. mandatory SCSI commands

  • Permissively licensed: Apache 2.0
  • Doubles as sample code for do-it-yourself-ers
  • Needed a prototype daemon in any case
slide-14
SLIDE 14

LIO AND TCMU 14

iSCSI commands

User Kernel disk.img /dev/vg0/vol0 Configuration Fabrics

tcm_fc iscsi

/dev/sdc

block file pscsi rtslib targetcli

LIO core

/sys/kernel/config/target

tcm-user

uio0 uio1 Backstores

tcmu-runner tape vmdk mmc smc glfs

N E W ! N E W !

slide-15
SLIDE 15

LIO AND TCMU 15

Handlers Need Confjg Info

  • All confjguration should still be through

standard LIO mechanisms (i.e. targetcli)

  • User backstore create includes a URL-like

“confjgstring” that gives handler and per-device handler-specifjc stuf

  • This is published in uio sysfs, and netlink

add_device message

  • Also has info to allow going from uio dev back

to matching LIO backstore

  • Handler needs attribs, block size, etc.
slide-16
SLIDE 16

LIO AND TCMU 16

Confjg tools need Info on Handlers Too!

  • Users should call “backstores/foo create x y z”

not “backstores/user <confjgstring>”

  • targetcli needs param and help strings for xyz
  • Verify backstore params are correct before

creating the device

  • Must be loosely coupled – no hard dependencies
  • n targetcli or tcmu-handler
  • Solution: D-Bus!
slide-17
SLIDE 17

LIO AND TCMU 17

iSCSI commands

User Kernel disk.img /dev/vg0/vol0 Configuration Fabrics

tcm_fc iscsi

/dev/sdc

block file pscsi rtslib targetcli

LIO core

/sys/kernel/config/target

tcm-user

uio0 uio1 Backstores

tcmu-runner tape vmdk mmc smc glfs

N E W ! N E W !

DBus

slide-18
SLIDE 18

LIO AND TCMU 18

The QEMU Question

  • QEMU has great support for many image

formats and other backstores

  • Can we reuse or integrate somehow?
  • How?
  • Build qemu handler code separately and

integrate as a tcmu-runner handler?

  • Extend qemu to implement TCMU directly, and

possibly also enable it to confjgure LIO exports?

  • Deferred for now
slide-19
SLIDE 19

LIO AND TCMU 19

Getting Involved

  • Give feedback!
  • agrover@redhat.com, target-devel@vger.kernel.org
  • Check out tcmu-runner:

https://github.com/agrover/tcmu-runner and its included sample handlers.

  • Use github PRs and issue tracking.
  • Much help needed, esp. QEMU hackers!
  • Start doing some TCMU performance benchmarking
  • Start thinking of interesting device types, userspace

libraries to use, or weird things to do with a SCSI command sandbox

slide-20
SLIDE 20

LIO AND TCMU 20

Thanks! Thanks! Questions? Questions?

slide-21
SLIDE 21

LIO AND TCMU 21

ISCSI commands

User Kernel disk.img /dev/vg0/vol0 Fabrics

tcm_fc iscsi

/dev/sdc

file pscsi

LIO core

/sys/kernel/config/target

tcm-user tcmu-runner block rtslib targetcli targetd liblvm lsmcli (libstoragemgmt)

Remote Local

N E W ! N E W !

slide-22
SLIDE 22

LIO AND TCMU 22

tcm-user tcmu-runner

/dev/uio0 /sys/class/uio/uio0 uio0

  • read to get

configuration

  • read (wait for cmd)
  • write (cmds done)
  • mmap (get SMR)

from LIO core Shared Memory Region Layout

(not to scale)

Mailbox Mailbox Command Ring Command Ring Data Area Data Area

cmdr_off cmdr_size cmd_head cmd_tail cmd_entry ...

  • pcode

iovec[] in/out data

User/Kernel Communication

...

slide-23
SLIDE 23

LIO AND TCMU 23

SCSI A set of standards for physically connecting and transferring data between computers and peripheral devices*

* http://en.wikipedia.org/wiki/SCSI

slide-24
SLIDE 24

LIO AND TCMU 24

SCSI target Initiator sends commands, Target handles them

slide-25
SLIDE 25

LIO AND TCMU 25

Multi-protocol SCSI target SCSI commands & data can be sent over many types of physical links and protocols. e.g:

  • SCSI Parallel Interface (original)
  • iSCSI (over TCP/IP)
  • SAS (over SATA cables)
  • Fibre Channel (over FCP)
  • FCoE (over Ethernet)
  • SRP (over Infiniband)
  • SBP-2 (over Firewire)
slide-26
SLIDE 26

LIO AND TCMU 26

tgtd

iSCSI commands from initiator

User Kernel

Backed by local files

  • r block devices

libglfs librbd

N E W ! N E W !

disk.img /dev/vg0/vol0

tgtadm

Configuration

slide-27
SLIDE 27

LIO AND TCMU 27

ISCSI commands

User Kernel disk.img /dev/vg0/vol0 Fabrics

tcm_fc iscsi

/dev/sdc

file pscsi

LIO core

/sys/kernel/config/target

tcm-user tcmu-runner

librbd libglfs qemu-lio-tcmu

block

qcow2 vmdk vdi rbd* glfs*

  • thers

C

  • m

i n g S

  • n

! C

  • m

i n g S

  • n

!