THE NEXT GENERATION CERTAINTY IN SHARED STORAGE ENVIRONMENTS Adam - - PowerPoint PPT Presentation

▶
the next generation
SMART_READER_LITE
LIVE PREVIEW

THE NEXT GENERATION CERTAINTY IN SHARED STORAGE ENVIRONMENTS Adam - - PowerPoint PPT Presentation

Mar 03, 2023 411 likes •738 views

THE NEXT GENERATION CERTAINTY IN SHARED STORAGE ENVIRONMENTS Adam Litke - alitke@redhat.com Senior Soware Engineer - Red Hat FOSDEM 2017 - 04 February 2017 The Next Generation - FOSDEM 04.02.2017 AGENDA oVirt shared storage architecture

slide-1
SLIDE 1

The Next Generation - FOSDEM 04.02.2017

THE NEXT GENERATION

CERTAINTY IN SHARED STORAGE ENVIRONMENTS

Adam Litke - Senior Soware Engineer - Red Hat FOSDEM 2017 - 04 February 2017 alitke@redhat.com

slide-2
SLIDE 2

The Next Generation - FOSDEM 04.02.2017

AGENDA

  • Virt shared storage architecture

😲 MAYHEM! 😍 Order. Examples

slide-3
SLIDE 3

The Next Generation - FOSDEM 04.02.2017

OVIRT SHARED STORAGE

NFS

Gluster Gluster

iSCSI

Ceph

VM

x 1000s x 100s x 1

Engine

Fibre Channel

slide-4
SLIDE 4

The Next Generation - FOSDEM 04.02.2017

OVIRT IMAGE

Volumes

A B C D

slide-5
SLIDE 5

The Next Generation - FOSDEM 04.02.2017

OVIRT VOLUME

Metadata Lease Data

slide-6
SLIDE 6

The Next Generation - FOSDEM 04.02.2017

STORAGE OPERATIONS

VM

Datapath Metadata

slide-7
SLIDE 7

The Next Generation - FOSDEM 04.02.2017

STORAGE JOBS

Encapsulate a singular storage operation on a host Engine selects a host and submits the job Host performs work asynchronously Engine polls host for job status Job status only available on host while job is active

slide-8
SLIDE 8

The Next Generation - FOSDEM 04.02.2017

THE WORLD IS A CHAOTIC PLACE

Power outages Network outages Hardware failure Soware bugs

slide-9
SLIDE 9

The Next Generation - FOSDEM 04.02.2017

FAILED HOST

Engine

Host A Host B

Vol 1 Vol 2 Vol 3

Storage

slide-10
SLIDE 10

The Next Generation - FOSDEM 04.02.2017

RESTORING ORDER

Determine the status of outstanding storage jobs The answers to all questions must come from storage Wait or abort active jobs on unresponsive hosts Check final status of jobs that have ended

slide-11
SLIDE 11

The Next Generation - FOSDEM 04.02.2017

VOLUME LEASES

Implemented using Lockspace is on shared storage alongside the volumes A lease grants a host exclusive access to a volume Failing hosts will be fenced by Sanlock if they hold leases Sanlock

Metadata Lease Data

slide-12
SLIDE 12

The Next Generation - FOSDEM 04.02.2017

VOLUME GENERATIONS

Monotonically increasing value Stored in volume metadata area Changeable only while holding the volume lease Allows sequencing of storage jobs

Metadata Lease Data

slide-13
SLIDE 13

The Next Generation - FOSDEM 04.02.2017

STORAGE JOB STRUCTURE

  • 1. Acquire volume lease
  • 2. Validate volume generation
  • 3. Do work
  • 4. Increment volume generation
  • 5. Release volume lease
slide-14
SLIDE 14

The Next Generation - FOSDEM 04.02.2017

EXAMPLE: NORMAL FLOW

slide-15
SLIDE 15

The Next Generation - FOSDEM 04.02.2017

SCHEDULE JOB

Lease

Engine

Host A Host B

Job Job

Vol 1 Vol 2 Vol 3

Storage

slide-16
SLIDE 16

The Next Generation - FOSDEM 04.02.2017

ACQUIRE VOLUME LEASE

Engine

Host A Host B

Job Job

Vol 1 Vol 2 Vol 3

Storage

Lease

slide-17
SLIDE 17

The Next Generation - FOSDEM 04.02.2017

VALIDATE VOLUME GENERATION

Engine

Host A Host B

Job Job

Vol 1 Vol 2 Vol 3

Storage

Lease

1 1

slide-18
SLIDE 18

The Next Generation - FOSDEM 04.02.2017

WRITE VOLUME

Engine

Host A Host B

Job Job

Vol 1 Vol 2 Vol 3

Storage

Lease

!

50%

slide-19
SLIDE 19

The Next Generation - FOSDEM 04.02.2017

INCREMENT VOLUME GENERATION

Engine

Host A Host B

Job Job

Vol 1 Vol 2 Vol 3

Storage

Lease

2

slide-20
SLIDE 20

The Next Generation - FOSDEM 04.02.2017

RELEASE VOLUME LEASE

Lease

Engine

Host A Host B

Job Job

Vol 2 Vol 3

Storage

Vol 1

slide-21
SLIDE 21

The Next Generation - FOSDEM 04.02.2017 The Next Generation - FOSDEM 04.02.2017

DONE EVENT

Lease

Engine

Host A Host B

Job Job

Vol 2 Vol 3

Storage

Vol 1

!

slide-22
SLIDE 22

The Next Generation - FOSDEM 04.02.2017

SCENARIO: UNRESPONSIVE HOST

slide-23
SLIDE 23

The Next Generation - FOSDEM 04.02.2017

JOB SCHEDULED ON HOST A

Engine

Host A Host B

Job Job

Vol 1 Vol 2 Vol 3

Storage 1

slide-24
SLIDE 24

The Next Generation - FOSDEM 04.02.2017

HOST A BECOMES UNRESPONSIVE

Engine

Host B

Vol 2 Vol 3

Storage

???

Host A

Lease

?

Vol 1

?

slide-25
SLIDE 25

The Next Generation - FOSDEM 04.02.2017

VOLUME RECONNAISSANCE

Special storage job that resolves volume status Checks if a job is running Option to use sanlock fencing to free the lease If volume is free, uses generation to check status Increments generation to preempt pending jobs

slide-26
SLIDE 26

The Next Generation - FOSDEM 04.02.2017

SELECT ANY AVAILABLE HOST

Engine

Host B

Vol 2 Vol 3

Storage

???

Host A

Lease

?

Vol 1

?

Job Job

1

slide-27
SLIDE 27

The Next Generation - FOSDEM 04.02.2017

ACQUIRE OR REQUEST LEASE

Host B

Job Job

Vol 1

Lease

!

Job Running

Host B

Job Job

Vol 1

Lease

...

  • r
slide-28
SLIDE 28

The Next Generation - FOSDEM 04.02.2017

COMPARE AND BUMP GENERATION

Host B

Job Job

Vol 1

Host B

Job Job

Vol 1

!

Job Succeeded

  • r

Vol 1

1 2

!

Job Failed

1 2 1

Lease Lease

slide-29
SLIDE 29

The Next Generation - FOSDEM 04.02.2017 The Next Generation - FOSDEM 04.02.2017

PREEMPTED JOB

Engine

Host A Host B

Job Job

Vol 1 Vol 2 Vol 3

Storage 1 2

Lease

slide-30
SLIDE 30

The Next Generation - FOSDEM 04.02.2017

FUTURE WORK

Shared lease support Parallel job scheduling Integrate with VM leases

slide-31
SLIDE 31

The Next Generation - FOSDEM 04.02.2017

JOIN US!

http://www.ovirt.org irc://irc.oc.net/ovirt http://lists.ovirt.org/mailman/listinfo/devel http://lists.ovirt.org/mailman/listinfo/users

slide-32
SLIDE 32

The Next Generation - FOSDEM 04.02.2017

QUESTIONS?