High Availability with No Split Brains! Arik Hadas Principal - - PowerPoint PPT Presentation

▶
high availability with no split brains
SMART_READER_LITE
LIVE PREVIEW

High Availability with No Split Brains! Arik Hadas Principal - - PowerPoint PPT Presentation

Aug 04, 2023 575 likes •1.14k views

High Availability with No Split Brains! Arik Hadas Principal Software Engineer Red Hat 27/01/2018 DevConf.cz, January 2018 Virtual Data Center Physical Servers DevConf.cz, January 2018 Virtual Data Center Virtual Machines

slide-1
SLIDE 1

DevConf.cz, January 2018

High Availability with No Split Brains!

Arik Hadas Principal Software Engineer Red Hat 27/01/2018

slide-2
SLIDE 2

DevConf.cz, January 2018

Virtual Data Center – Physical Servers

slide-3
SLIDE 3

DevConf.cz, January 2018

Virtual Data Center – Virtual Machines

slide-4
SLIDE 4

DevConf.cz, January 2018

Virtual Data Center - Applications

slide-5
SLIDE 5

DevConf.cz, January 2018

Some Applications are More Critical

slide-6
SLIDE 6

DevConf.cz, January 2018

High Availability - Application-Level

slide-7
SLIDE 7

DevConf.cz, January 2018

High Availability - Application-Level

slide-8
SLIDE 8

DevConf.cz, January 2018

High Availability - Application-Level

  • Higher resource consumption
  • More responsibility on the application
  • Backup starts in a different environment

– Different IP address(es) – Different disk(s)

slide-9
SLIDE 9

DevConf.cz, January 2018

High Availability - VM-Level

slide-10
SLIDE 10

DevConf.cz, January 2018

High Availability - VM-Level

slide-11
SLIDE 11

DevConf.cz, January 2018

High Availability - VM-Level

  • More efficient resource consumption
  • Implemented at the infrastructure level
  • VM always start in the same environment

– Same IP address(es) – Same disk(s)

slide-12
SLIDE 12

DevConf.cz, January 2018

Central Monitoring Unit

slide-13
SLIDE 13

DevConf.cz, January 2018

Fault Detection

HA VM went down!

slide-14
SLIDE 14

DevConf.cz, January 2018

Automatic Restart

Restart the VM

slide-15
SLIDE 15

DevConf.cz, January 2018

Automatic Restart – Not That Simple

Restart the VM What if:

– Inaccessible resources – VM is locked – VM is being

intentionally shut down

slide-16
SLIDE 16

DevConf.cz, January 2018

Automatic Restart – Not That Simple

What if:

– Inaccessible resources – VM is locked – VM is being

intentionally shut down

AutoStartVmsRunner

https://github.com/oVirt/ovirt-engine/blob/master/backend /manager/modules/bll/src/main/java/org/ovirt/engine/core/ bll/AutoStartVmsRunner.java

slide-17
SLIDE 17

DevConf.cz, January 2018

AutoStartVmsRunner

Lock VM Should Restart? Run No More Tries

slide-18
SLIDE 18

DevConf.cz, January 2018

Fault Detection – Even More Complex

slide-19
SLIDE 19

DevConf.cz, January 2018

Fault Detection – Even More Complex

slide-20
SLIDE 20

DevConf.cz, January 2018

Fault Detection – Even More Complex

Is the left server alive?

slide-21
SLIDE 21

DevConf.cz, January 2018

Fault Detection – Even More Complex

Is the HA VM running?

slide-22
SLIDE 22

DevConf.cz, January 2018

Fault Detection – Manual Confjrmation

The server has been rebooted

slide-23
SLIDE 23

DevConf.cz, January 2018

Fault Detection – Manual Confjrmation

Restart the VM

slide-24
SLIDE 24

DevConf.cz, January 2018

Fault Detection – Manual Confjrmation

  • Slow
  • Error-prone

– Mistakes may lead to a split-brain

slide-25
SLIDE 25

DevConf.cz, January 2018

Split Brain of Virtual Machines

A scenario in which several instances

  • f the same VM run simultaneously
slide-26
SLIDE 26

DevConf.cz, January 2018

Split Brain Due to a False Confjrmation

May lead to data corruption!

slide-27
SLIDE 27

DevConf.cz, January 2018

Split Brains May Happen Due to Bugs

Only the right VM is reported

slide-28
SLIDE 28

DevConf.cz, January 2018

Split Brains May Happen Due to Bugs

Restart the left VM

slide-29
SLIDE 29

DevConf.cz, January 2018

VM Leases: Our Solution to Split Brains

slide-30
SLIDE 30

DevConf.cz, January 2018

VM Leases: Our Solution to Split Brains

VM will not start while its lease exists

slide-31
SLIDE 31

DevConf.cz, January 2018

VM Lease Creation

slide-32
SLIDE 32

DevConf.cz, January 2018

VM Lease Creation

slide-33
SLIDE 33

DevConf.cz, January 2018

VM Lease Creation

“Create a VM Lease for VM X in storage domain Y” SPM

slide-34
SLIDE 34

DevConf.cz, January 2018

VM Lease Creation

“Create a Lease X in lockspace Y” “Create a VM Lease for VM X in storage domain Y” SPM

slide-35
SLIDE 35

DevConf.cz, January 2018

VM Lease Creation

“Path P to xleases volume and Lease

  • ffset O”

“Create a VM Lease for VM X in storage domain Y” “Create a Lease X in lockspace Y” SPM

slide-36
SLIDE 36

DevConf.cz, January 2018

xleases volume

  • Sanlock does not manage leases allocation
  • Volume layout:
  • Same format in block and file storage
  • Deep Dive - VM leases (youtube)

lockspace index master lease user lease 1 user lease 2 ....

slide-37
SLIDE 37

DevConf.cz, January 2018

Running a VM with a Lease

<domain type='kvm' id='6'> <name>fedora8</name> ... skipped ... <devices> ... skipped ...

<lease>

<lockspace>571184ae-79da-41fb-a3fb-c3117991abae</lockspace>

<key>cbd783e4-45f8-4b51-93ca-4460d4dad772</key> <target path='/rhev/data-center/mnt/10.35.1.90:_srv_Default/571184ae-

79da-41fb-a3fb-c3117991abae/dom_md/xleases' offset='3145728'/>

</lease> ... skipped ... </domain>

slide-38
SLIDE 38

DevConf.cz, January 2018

Running a VM with a Lease

Domain XML with Lease Acquires the Lease using Sanlock Lease

slide-39
SLIDE 39

DevConf.cz, January 2018

Non-Responsive Host Treatment

slide-40
SLIDE 40

DevConf.cz, January 2018

Non-Responsive Host Treatment

slide-41
SLIDE 41

DevConf.cz, January 2018

Non-Responsive Host Treatment

60+ sec of grace period

slide-42
SLIDE 42

DevConf.cz, January 2018

Non-Responsive Host Treatment

Fence (power management )

slide-43
SLIDE 43

DevConf.cz, January 2018

Non-Responsive Host Treatment

Restart VMs with a Lease

slide-44
SLIDE 44

DevConf.cz, January 2018

(1) Non-Responsive Host + VM is Down

Restart VMs with a Lease

slide-45
SLIDE 45

DevConf.cz, January 2018

(1) Non-Responsive Host + VM is Down

VM starts on another host

slide-46
SLIDE 46

DevConf.cz, January 2018

(2) Non-Responsive Host + VM is UP

Restart VMs with a Lease

slide-47
SLIDE 47

DevConf.cz, January 2018

(2) Non-Responsive Host + VM is UP

Restart VMs with a Lease

slide-48
SLIDE 48

DevConf.cz, January 2018

Disconnection From Storage Device

slide-49
SLIDE 49

DevConf.cz, January 2018

Disconnection From Storage Device (1)

(1) Lease expires

slide-50
SLIDE 50

DevConf.cz, January 2018

Disconnection From Storage Device (1)

(2) VM is terminated (1) Lease expires

slide-51
SLIDE 51

DevConf.cz, January 2018

Disconnection From Storage Device (2)

(1) VM is paused (2) Lease is released

slide-52
SLIDE 52

DevConf.cz, January 2018

Summary

  • VM Lease – an important new element

– Prevents split-brains – Enables automatic restart of unreported VMs

  • Available since oVirt 4.1

– Polished in oVirt 4.2

  • Possible future enhancements:

– May be used to restart paused VMs – Move together with the bootable disk

slide-53
SLIDE 53

DevConf.cz, January 2018

THANK YOU!

http://www.ovirt.org ahadas@redhat.com ahadas@irc.oftc.net#ovirt