CTDB database vacuuming for geniuses Amitay Isaacs amitay@samba.org - - PowerPoint PPT Presentation

▶
ctdb database vacuuming for geniuses
SMART_READER_LITE
LIVE PREVIEW

CTDB database vacuuming for geniuses Amitay Isaacs amitay@samba.org - - PowerPoint PPT Presentation

May 27, 2023 417 likes •1.59k views

CTDB database vacuuming for geniuses Amitay Isaacs amitay@samba.org Samba Team IBM (Australia Development Labs, Linux Technology Center) Amitay Isaacs CTDB database vacuuming for geniuses CTDB Project Motivation: Support for clustered Samba

slide-1
SLIDE 1

CTDB database vacuuming for geniuses

Amitay Isaacs amitay@samba.org

Samba Team IBM (Australia Development Labs, Linux Technology Center)

Amitay Isaacs CTDB database vacuuming for geniuses

slide-2
SLIDE 2

CTDB Project

Motivation: Support for clustered Samba Multiple nodes active simultaneously Communication between nodes (heartbeat, failover) Distributed databases between nodes Features: Volatile and Persistent databases Cluster-side messaging for Samba IP failover and load balancing Service monitoring Community: https://wiki.samba.org git://git.samba.org/samba.git

Amitay Isaacs CTDB database vacuuming for geniuses

slide-3
SLIDE 3

Overview

Database models Volatile database

Data model Vacuuming Recovery

Zombie outbreak

Resurrection of the dead Keeping ’em dead

Restructure

Amitay Isaacs CTDB database vacuuming for geniuses

slide-4
SLIDE 4

Database models

Volatile Persistent distributed data replicated data single copy multiple copies data loss on failure loss-less per-node per-db per-chain mutex clusterwide per-db mutex disk backed disk backed cluster-wide traverse local traverse shared access client-server access

Amitay Isaacs CTDB database vacuuming for geniuses

slide-5
SLIDE 5

Volatile database

Amitay Isaacs CTDB database vacuuming for geniuses

slide-6
SLIDE 6

Volatile database – data model – record creation

Node1 Node2 Node3

Amitay Isaacs CTDB database vacuuming for geniuses

slide-7
SLIDE 7

Volatile database – data model – record creation

Node1 Node2 Node3

Node1: Samba requests record1

Amitay Isaacs CTDB database vacuuming for geniuses

slide-8
SLIDE 8

Volatile database – data model – record creation

Node1 Node2 Node3

Node1: Samba requests record1 Node1: Where is record1?

Amitay Isaacs CTDB database vacuuming for geniuses

slide-9
SLIDE 9

Volatile database – data model – record creation

Node1 Node2 Node3

LMASTER

Node1: Samba requests record1 Node1: Where is record1?

Amitay Isaacs CTDB database vacuuming for geniuses

slide-10
SLIDE 10

Volatile database – data model – record creation

Node1 Node2 Node3

Node1: Samba requests record1 Node1: Where is record1? Node1: Request for record1

Amitay Isaacs CTDB database vacuuming for geniuses

slide-11
SLIDE 11

Volatile database – data model – record creation

Node1 Node2 Node3

Node1: Samba requests record1 Node1: Where is record1? Node1: Request for record1 Node2: Where is record1?

Amitay Isaacs CTDB database vacuuming for geniuses

slide-12
SLIDE 12

Volatile database – data model – record creation

Node1 Node2 Node3

LMASTER

Node1: Samba requests record1 Node1: Where is record1? Node1: Request for record1 Node2: Where is record1?

Amitay Isaacs CTDB database vacuuming for geniuses

slide-13
SLIDE 13

Volatile database – data model – record creation

Node1 Node2 Node3

LMASTER DMASTER

rsn:1

Node1: Samba requests record1 Node1: Where is record1? Node1: Request for record1 Node2: Where is record1? Node2: Create empty record1

Amitay Isaacs CTDB database vacuuming for geniuses

slide-14
SLIDE 14

Volatile database – data model – record creation

Node1 Node2 Node3

LMASTER DMASTER

rsn:1

Node1: Samba requests record1 Node1: Where is record1? Node1: Request for record1 Node2: Where is record1? Node2: Create empty record1 Node2: Migrate record1 to Node1

Amitay Isaacs CTDB database vacuuming for geniuses

slide-15
SLIDE 15

Volatile database – data model – record creation

Node1 Node2 Node3

LMASTER DMASTER

rsn:1 rsn:3

Node1: Samba requests record1 Node1: Where is record1? Node1: Request for record1 Node2: Where is record1? Node2: Create empty record1 Node2: Migrate record1 to Node1 Node1: Receive record1

Amitay Isaacs CTDB database vacuuming for geniuses

slide-16
SLIDE 16

Volatile database – data model – record creation

Node1 Node2 Node3

LMASTER DMASTER

rsn:1 rsn:3

Node1: Samba requests record1 Node1: Where is record1? Node1: Request for record1 Node2: Where is record1? Node2: Create empty record1 Node2: Migrate record1 to Node1 Node1: Receive record1 Node1: Send reply to samba

Amitay Isaacs CTDB database vacuuming for geniuses

slide-17
SLIDE 17

Volatile database – data model – local access

Node1 Node2 Node3

LMASTER DMASTER

rsn:1 rsn:3

Amitay Isaacs CTDB database vacuuming for geniuses

slide-18
SLIDE 18

Volatile database – data model – local access

Node1 Node2 Node3

LMASTER DMASTER

rsn:1 rsn:3

Node1: Samba needs record1

Amitay Isaacs CTDB database vacuuming for geniuses

slide-19
SLIDE 19

Volatile database – data model – local access

Node1 Node2 Node3

LMASTER DMASTER

rsn:1 rsn:3

Node1: Samba needs record1 Node1: Samba checks if record1 is local

Amitay Isaacs CTDB database vacuuming for geniuses

slide-20
SLIDE 20

Volatile database – data model – local access

Node1 Node2 Node3

LMASTER DMASTER

rsn:1 rsn:3

Node1: Samba needs record1 Node1: Samba checks if record1 is local Node1: Samba uses record1

Amitay Isaacs CTDB database vacuuming for geniuses

slide-21
SLIDE 21

Volatile database – data model – local access

Node1 Node2 Node3

LMASTER DMASTER

rsn:1 rsn:3

Node1: Samba needs record1 Node1: Samba checks if record1 is local Node1: Samba uses record1 Node1: Samba uses record1

Amitay Isaacs CTDB database vacuuming for geniuses

slide-22
SLIDE 22

Volatile database – data model – local access

Node1 Node2 Node3

LMASTER DMASTER

rsn:1 rsn:3

Node1: Samba needs record1 Node1: Samba checks if record1 is local Node1: Samba uses record1 Node1: Samba uses record1 Node1: Samba uses record1

Amitay Isaacs CTDB database vacuuming for geniuses

slide-23
SLIDE 23

Volatile database – data model – local access

Node1 Node2 Node3

LMASTER DMASTER

rsn:1 rsn:3

Node1: Samba needs record1 Node1: Samba checks if record1 is local Node1: Samba uses record1 Node1: Samba uses record1 Node1: Samba uses record1 CTDB is not involved

Amitay Isaacs CTDB database vacuuming for geniuses

slide-24
SLIDE 24

Volatile database – data model – remote access

Node1 Node2 Node3

LMASTER DMASTER

rsn:1 rsn:3

Amitay Isaacs CTDB database vacuuming for geniuses

slide-25
SLIDE 25

Volatile database – data model – remote access

Node1 Node2 Node3

LMASTER DMASTER

rsn:1 rsn:3

Node3: Samba requests record1

Amitay Isaacs CTDB database vacuuming for geniuses

slide-26
SLIDE 26

Volatile database – data model – remote access

Node1 Node2 Node3

LMASTER DMASTER

rsn:1 rsn:3

Node3: Samba requests record1 Node3: Where is record1?

Amitay Isaacs CTDB database vacuuming for geniuses

slide-27
SLIDE 27

Volatile database – data model – remote access

Node1 Node2 Node3

LMASTER DMASTER

rsn:1 rsn:3

Node3: Samba requests record1 Node3: Where is record1? Node3: Request for record1

Amitay Isaacs CTDB database vacuuming for geniuses

slide-28
SLIDE 28

Volatile database – data model – remote access

Node1 Node2 Node3

LMASTER DMASTER

rsn:1 rsn:3

Node3: Samba requests record1 Node3: Where is record1? Node3: Request for record1 Node2: Where is record1?

Amitay Isaacs CTDB database vacuuming for geniuses

slide-29
SLIDE 29

Volatile database – data model – remote access

Node1 Node2 Node3

LMASTER DMASTER

rsn:1 rsn:3

Node3: Samba requests record1 Node3: Where is record1? Node3: Request for record1 Node2: Where is record1? Node2: Redirect request for record1

Amitay Isaacs CTDB database vacuuming for geniuses

slide-30
SLIDE 30

Volatile database – data model – remote access

Node1 Node2 Node3

LMASTER DMASTER

rsn:1 rsn:3

Node3: Samba requests record1 Node3: Where is record1? Node3: Request for record1 Node2: Where is record1? Node2: Redirect request for record1 Node2: Where is record1?

Amitay Isaacs CTDB database vacuuming for geniuses

slide-31
SLIDE 31

Volatile database – data model – remote access

Node1 Node2 Node3

LMASTER

rsn:1 rsn:3

Node3: Samba requests record1 Node3: Where is record1? Node3: Request for record1 Node2: Where is record1? Node2: Redirect request for record1 Node2: Where is record1? Node1: Migrate record1 to Node2

Amitay Isaacs CTDB database vacuuming for geniuses

slide-32
SLIDE 32

Volatile database – data model – remote access

Node1 Node2 Node3

LMASTER

rsn:3 rsn:3

Node3: Samba requests record1 Node3: Where is record1? Node3: Request for record1 Node2: Where is record1? Node2: Redirect request for record1 Node2: Where is record1? Node1: Migrate record1 to Node2 Node2: Receive record1 and update location

Amitay Isaacs CTDB database vacuuming for geniuses

slide-33
SLIDE 33

Volatile database – data model – remote access

Node1 Node2 Node3

LMASTER

rsn:3 rsn:3

Node3: Samba requests record1 Node3: Where is record1? Node3: Request for record1 Node2: Where is record1? Node2: Redirect request for record1 Node2: Where is record1? Node1: Migrate record1 to Node2 Node2: Receive record1 and update location Node2: Migrate record1 to Node3

Amitay Isaacs CTDB database vacuuming for geniuses

slide-34
SLIDE 34

Volatile database – data model – remote access

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Node3: Samba requests record1 Node3: Where is record1? Node3: Request for record1 Node2: Where is record1? Node2: Redirect request for record1 Node2: Where is record1? Node1: Migrate record1 to Node2 Node2: Receive record1 and update location Node2: Migrate record1 to Node3 Node3: Receive record1

Amitay Isaacs CTDB database vacuuming for geniuses

slide-35
SLIDE 35

Volatile database – data model – remote access

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Node3: Samba requests record1 Node3: Where is record1? Node3: Request for record1 Node2: Where is record1? Node2: Redirect request for record1 Node2: Where is record1? Node1: Migrate record1 to Node2 Node2: Receive record1 and update location Node2: Migrate record1 to Node3 Node3: Receive record1 Node3: Send reply to samba

Amitay Isaacs CTDB database vacuuming for geniuses

slide-36
SLIDE 36

Volatile database – data model – record deletion

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Amitay Isaacs CTDB database vacuuming for geniuses

slide-37
SLIDE 37

Volatile database – data model – record deletion

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Node3: Samba deletes record1

Amitay Isaacs CTDB database vacuuming for geniuses

slide-38
SLIDE 38

Volatile database – data model – record deletion

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Node3: Samba deletes record1 Record is actually not deleted

Amitay Isaacs CTDB database vacuuming for geniuses

slide-39
SLIDE 39

Volatile database – data model – record deletion

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Node3: Samba deletes record1 Record is actually not deleted

There are copies of the record1 on all nodes

Amitay Isaacs CTDB database vacuuming for geniuses

slide-40
SLIDE 40

Volatile database – data model – record deletion

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Node3: Samba deletes record1 Record is actually not deleted

There are copies of the record1 on all nodes How to delete all copies?

Amitay Isaacs CTDB database vacuuming for geniuses

slide-41
SLIDE 41

Volatile database – data model – record deletion

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Node3: Samba deletes record1 Record is actually not deleted

There are copies of the record1 on all nodes How to delete all copies?

Node3: Record is marked to be deleted

Amitay Isaacs CTDB database vacuuming for geniuses

slide-42
SLIDE 42

Volatile database – data model – record deletion

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Node3: Samba deletes record1 Record is actually not deleted

There are copies of the record1 on all nodes How to delete all copies?

Node3: Record is marked to be deleted Vacuuming

Amitay Isaacs CTDB database vacuuming for geniuses

slide-43
SLIDE 43

Volatile database – vacuuming

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Amitay Isaacs CTDB database vacuuming for geniuses

slide-44
SLIDE 44

Volatile database – vacuuming

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Node3: Record is marked for deletion

Amitay Isaacs CTDB database vacuuming for geniuses

slide-45
SLIDE 45

Volatile database – vacuuming

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Node3: Record is marked for deletion Node3: Vacuuming: Send the record to lmaster

Amitay Isaacs CTDB database vacuuming for geniuses

slide-46
SLIDE 46

Volatile database – vacuuming

Node1 Node2 Node3

LMASTER DMASTER

rsn:7 rsn:3 rsn:5

Node3: Record is marked for deletion Node3: Vacuuming: Send the record to lmaster Node2: Receive record & mark for deletion

Amitay Isaacs CTDB database vacuuming for geniuses

slide-47
SLIDE 47

Volatile database – vacuuming

Node1 Node2 Node3

LMASTER DMASTER

rsn:7 rsn:3 rsn:5

Node3: Record is marked for deletion Node3: Vacuuming: Send the record to lmaster Node2: Receive record & mark for deletion Node2: Vacuuming: Delete record from other nodes

Amitay Isaacs CTDB database vacuuming for geniuses

slide-48
SLIDE 48

Volatile database – vacuuming

Node1 Node2 Node3

LMASTER DMASTER

rsn:7

Node3: Record is marked for deletion Node3: Vacuuming: Send the record to lmaster Node2: Receive record & mark for deletion Node2: Vacuuming: Delete record from other nodes

Amitay Isaacs CTDB database vacuuming for geniuses

slide-49
SLIDE 49

Volatile database – vacuuming

Node1 Node2 Node3

LMASTER DMASTER

rsn:7

Node3: Record is marked for deletion Node3: Vacuuming: Send the record to lmaster Node2: Receive record & mark for deletion Node2: Vacuuming: Delete record from other nodes Node2: Vacuuming: Delete record locally

Amitay Isaacs CTDB database vacuuming for geniuses

slide-50
SLIDE 50

Volatile database – vacuuming

Node1 Node2 Node3

Node3: Record is marked for deletion Node3: Vacuuming: Send the record to lmaster Node2: Receive record & mark for deletion Node2: Vacuuming: Delete record from other nodes Node2: Vacuuming: Delete record locally

Amitay Isaacs CTDB database vacuuming for geniuses

slide-51
SLIDE 51

Volatile database – recovery

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Amitay Isaacs CTDB database vacuuming for geniuses

slide-52
SLIDE 52

Volatile database – recovery

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Node1: Starts database recovery

Amitay Isaacs CTDB database vacuuming for geniuses

slide-53
SLIDE 53

Volatile database – recovery

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Recovery Helper

Node1: Starts database recovery

Amitay Isaacs CTDB database vacuuming for geniuses

slide-54
SLIDE 54

Volatile database – recovery

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Recovery Helper

Node1: Starts database recovery Recovery: Freeze database

Amitay Isaacs CTDB database vacuuming for geniuses

slide-55
SLIDE 55

Volatile database – recovery

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Recovery Helper

Node1: Starts database recovery Recovery: Freeze database Recovery: Collect records from Node1

Amitay Isaacs CTDB database vacuuming for geniuses

slide-56
SLIDE 56

Volatile database – recovery

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Recovery Helper

rsn:3

Node1: Starts database recovery Recovery: Freeze database Recovery: Collect records from Node1

Amitay Isaacs CTDB database vacuuming for geniuses

slide-57
SLIDE 57

Volatile database – recovery

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Recovery Helper

rsn:3

Node1: Starts database recovery Recovery: Freeze database Recovery: Collect records from Node1 Recovery: Collect records from Node2

Amitay Isaacs CTDB database vacuuming for geniuses

slide-58
SLIDE 58

Volatile database – recovery

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Recovery Helper

rsn:3

Node1: Starts database recovery Recovery: Freeze database Recovery: Collect records from Node1 Recovery: Collect records from Node2

Amitay Isaacs CTDB database vacuuming for geniuses

slide-59
SLIDE 59

Volatile database – recovery

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Recovery Helper

rsn:3

Node1: Starts database recovery Recovery: Freeze database Recovery: Collect records from Node1 Recovery: Collect records from Node2 Recovery: Collect records from Node3

Amitay Isaacs CTDB database vacuuming for geniuses

slide-60
SLIDE 60

Volatile database – recovery

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Recovery Helper

rsn:5

Node1: Starts database recovery Recovery: Freeze database Recovery: Collect records from Node1 Recovery: Collect records from Node2 Recovery: Collect records from Node3

Amitay Isaacs CTDB database vacuuming for geniuses

slide-61
SLIDE 61

Volatile database – recovery

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Recovery Helper

rsn:5

Node1: Starts database recovery Recovery: Freeze database Recovery: Collect records from Node1 Recovery: Collect records from Node2 Recovery: Collect records from Node3 Recovery: Push the records to all the nodes

Amitay Isaacs CTDB database vacuuming for geniuses

slide-62
SLIDE 62

Volatile database – recovery

Node1 Node2 Node3

LMASTER DMASTER

rsn:5 rsn:5 rsn:5

Recovery Helper

Node1: Starts database recovery Recovery: Freeze database Recovery: Collect records from Node1 Recovery: Collect records from Node2 Recovery: Collect records from Node3 Recovery: Push the records to all the nodes

Amitay Isaacs CTDB database vacuuming for geniuses

slide-63
SLIDE 63

Volatile database – recovery

Node1 Node2 Node3

LMASTER DMASTER

rsn:5 rsn:5 rsn:5

Recovery Helper

Node1: Starts database recovery Recovery: Freeze database Recovery: Collect records from Node1 Recovery: Collect records from Node2 Recovery: Collect records from Node3 Recovery: Push the records to all the nodes Recovery: Thaw database

Amitay Isaacs CTDB database vacuuming for geniuses

slide-64
SLIDE 64

Zombie outbreak

Amitay Isaacs CTDB database vacuuming for geniuses

slide-65
SLIDE 65

Resurrection of the dead – background

Amitay Isaacs CTDB database vacuuming for geniuses

slide-66
SLIDE 66

Resurrection of the dead – background

Inactive node(s)

Amitay Isaacs CTDB database vacuuming for geniuses

slide-67
SLIDE 67

Resurrection of the dead – background

Inactive node(s)

Node is stopped

Amitay Isaacs CTDB database vacuuming for geniuses

slide-68
SLIDE 68

Resurrection of the dead – background

Inactive node(s)

Node is stopped Node is banned

Amitay Isaacs CTDB database vacuuming for geniuses

slide-69
SLIDE 69

Resurrection of the dead – background

Inactive node(s)

Node is stopped Node is banned

What happens?

Amitay Isaacs CTDB database vacuuming for geniuses

slide-70
SLIDE 70

Resurrection of the dead – background

Inactive node(s)

Node is stopped Node is banned

What happens?

VNN map changes

Amitay Isaacs CTDB database vacuuming for geniuses

slide-71
SLIDE 71

Resurrection of the dead – background

Inactive node(s)

Node is stopped Node is banned

What happens?

VNN map changes . . . causing database recovery

Amitay Isaacs CTDB database vacuuming for geniuses

slide-72
SLIDE 72

Resurrection of the dead – background

Inactive node(s)

Node is stopped Node is banned

What happens?

VNN map changes . . . causing database recovery

What happens when node becomes active again?

Amitay Isaacs CTDB database vacuuming for geniuses

slide-73
SLIDE 73

Resurrection of the dead – background

Inactive node(s)

Node is stopped Node is banned

What happens?

VNN map changes . . . causing database recovery

What happens when node becomes active again?

VNN map changes . . . causing database recovery

Amitay Isaacs CTDB database vacuuming for geniuses

slide-74
SLIDE 74

Resurrection of the dead – example

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Amitay Isaacs CTDB database vacuuming for geniuses

slide-75
SLIDE 75

Resurrection of the dead – example

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Node3 becomes inactive

Amitay Isaacs CTDB database vacuuming for geniuses

slide-76
SLIDE 76

Resurrection of the dead – example

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Recovery Helper

Node3 becomes inactive Node1 starts recovery

Amitay Isaacs CTDB database vacuuming for geniuses

slide-77
SLIDE 77

Resurrection of the dead – example

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Recovery Helper

Node3 becomes inactive Node1 starts recovery Recovery: Collect records from Node1 & Node2

Amitay Isaacs CTDB database vacuuming for geniuses

slide-78
SLIDE 78

Resurrection of the dead – example

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Recovery Helper

rsn:3

Node3 becomes inactive Node1 starts recovery Recovery: Collect records from Node1 & Node2

Amitay Isaacs CTDB database vacuuming for geniuses

slide-79
SLIDE 79

Resurrection of the dead – example

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Recovery Helper

rsn:3

Node3 becomes inactive Node1 starts recovery Recovery: Collect records from Node1 & Node2 Recovery: Push the records to Node1 & Node2

Amitay Isaacs CTDB database vacuuming for geniuses

slide-80
SLIDE 80

Resurrection of the dead – example

Node1 Node2 Node3

LMASTER DMASTER DMASTER

rsn:3 rsn:3 rsn:5

Recovery Helper

rsn:3

Node3 becomes inactive Node1 starts recovery Recovery: Collect records from Node1 & Node2 Recovery: Push the records to Node1 & Node2

Amitay Isaacs CTDB database vacuuming for geniuses

slide-81
SLIDE 81

Resurrection of the dead – example

Node1 Node2 Node3

LMASTER DMASTER DMASTER

rsn:3 rsn:3 rsn:5

Node3 becomes inactive Node1 starts recovery Recovery: Collect records from Node1 & Node2 Recovery: Push the records to Node1 & Node2 Recovery done

Amitay Isaacs CTDB database vacuuming for geniuses

slide-82
SLIDE 82

Resurrection of the dead – example

Node1 Node2 Node3

LMASTER DMASTER DMASTER

rsn:3 rsn:3 rsn:5

Node3 becomes inactive Node1 starts recovery Recovery: Collect records from Node1 & Node2 Recovery: Push the records to Node1 & Node2 Recovery done Node2: Samba deletes record

Amitay Isaacs CTDB database vacuuming for geniuses

slide-83
SLIDE 83

Resurrection of the dead – example

Node1 Node2 Node3

DMASTER

rsn:5

Node3 becomes inactive Node1 starts recovery Recovery: Collect records from Node1 & Node2 Recovery: Push the records to Node1 & Node2 Recovery done Node2: Samba deletes record Node2: Vacuuming removes record

Amitay Isaacs CTDB database vacuuming for geniuses

slide-84
SLIDE 84

Resurrection of the dead – example

Node1 Node2 Node3

DMASTER

rsn:5

Node3 becomes inactive Node1 starts recovery Recovery: Collect records from Node1 & Node2 Recovery: Push the records to Node1 & Node2 Recovery done Node2: Samba deletes record Node2: Vacuuming removes record Node3 becomes active

Amitay Isaacs CTDB database vacuuming for geniuses

slide-85
SLIDE 85

Resurrection of the dead – example

Node1 Node2 Node3

DMASTER

rsn:5

Node3 becomes inactive Node1 starts recovery Recovery: Collect records from Node1 & Node2 Recovery: Push the records to Node1 & Node2 Recovery done Node2: Samba deletes record Node2: Vacuuming removes record Node3 becomes active Zombie record!

Amitay Isaacs CTDB database vacuuming for geniuses

slide-86
SLIDE 86

Keeping ’em dead

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

Amitay Isaacs CTDB database vacuuming for geniuses

slide-87
SLIDE 87

Keeping ’em dead

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

When Node3 becomes inactive,

Amitay Isaacs CTDB database vacuuming for geniuses

slide-88
SLIDE 88

Keeping ’em dead

Node1 Node2 Node3

LMASTER DMASTER

rsn:3 rsn:3 rsn:5

When Node3 becomes inactive, . . . invalidate all records

Amitay Isaacs CTDB database vacuuming for geniuses

slide-89
SLIDE 89

Keeping ’em dead

Node1 Node2 Node3

LMASTER DMASTER DMASTER

rsn:3 rsn:3 rsn:5

When Node3 becomes inactive, . . . invalidate all records . . . database recovery

Amitay Isaacs CTDB database vacuuming for geniuses

slide-90
SLIDE 90

Keeping ’em dead

Node1 Node2 Node3

DMASTER

rsn:5

When Node3 becomes inactive, . . . invalidate all records . . . database recovery Record is deleted and vacuumed

Amitay Isaacs CTDB database vacuuming for geniuses

slide-91
SLIDE 91

Keeping ’em dead

Node1 Node2 Node3

DMASTER

rsn:5

When Node3 becomes inactive, . . . invalidate all records . . . database recovery Record is deleted and vacuumed Node3 becomes active,

Amitay Isaacs CTDB database vacuuming for geniuses

slide-92
SLIDE 92

Keeping ’em dead

Node1 Node2 Node3

When Node3 becomes inactive, . . . invalidate all records . . . database recovery Record is deleted and vacuumed Node3 becomes active, . . . database recovery

Amitay Isaacs CTDB database vacuuming for geniuses

slide-93
SLIDE 93

Keeping ’em dead

Node1 Node2 Node3

When Node3 becomes inactive, . . . invalidate all records . . . database recovery Record is deleted and vacuumed Node3 becomes active, . . . database recovery Patches coming soon

Amitay Isaacs CTDB database vacuuming for geniuses

slide-94
SLIDE 94

Restructure

Amitay Isaacs CTDB database vacuuming for geniuses

slide-95
SLIDE 95

Existing issues with vacuuming

Amitay Isaacs CTDB database vacuuming for geniuses

slide-96
SLIDE 96

Existing issues with vacuuming

Vacuuming done in a child process

Amitay Isaacs CTDB database vacuuming for geniuses

slide-97
SLIDE 97

Existing issues with vacuuming

Vacuuming done in a child process

Forks a process from main ctdb daemon

Amitay Isaacs CTDB database vacuuming for geniuses

slide-98
SLIDE 98

Existing issues with vacuuming

Vacuuming done in a child process

Forks a process from main ctdb daemon Can be very expensive

Amitay Isaacs CTDB database vacuuming for geniuses

slide-99
SLIDE 99

Existing issues with vacuuming

Vacuuming done in a child process

Forks a process from main ctdb daemon Can be very expensive

Migrating deleted record to lmaster

Amitay Isaacs CTDB database vacuuming for geniuses

slide-100
SLIDE 100

Existing issues with vacuuming

Vacuuming done in a child process

Forks a process from main ctdb daemon Can be very expensive

Migrating deleted record to lmaster

CTDB daemon itself cannot start record migration

Amitay Isaacs CTDB database vacuuming for geniuses

slide-101
SLIDE 101

Existing issues with vacuuming

Vacuuming done in a child process

Forks a process from main ctdb daemon Can be very expensive

Migrating deleted record to lmaster

CTDB daemon itself cannot start record migration Migration is triggered from a client

Amitay Isaacs CTDB database vacuuming for geniuses

slide-102
SLIDE 102

Existing issues with vacuuming

Vacuuming done in a child process

Forks a process from main ctdb daemon Can be very expensive

Migrating deleted record to lmaster

CTDB daemon itself cannot start record migration Migration is triggered from a client Hook vacuuming migration in recovery daemon (ugh!)

Amitay Isaacs CTDB database vacuuming for geniuses

slide-103
SLIDE 103

Existing issues with vacuuming

Vacuuming done in a child process

Forks a process from main ctdb daemon Can be very expensive

Migrating deleted record to lmaster

CTDB daemon itself cannot start record migration Migration is triggered from a client Hook vacuuming migration in recovery daemon (ugh!)

Deleting empty record from other nodes

Amitay Isaacs CTDB database vacuuming for geniuses

slide-104
SLIDE 104

Existing issues with vacuuming

Vacuuming done in a child process

Forks a process from main ctdb daemon Can be very expensive

Migrating deleted record to lmaster

CTDB daemon itself cannot start record migration Migration is triggered from a client Hook vacuuming migration in recovery daemon (ugh!)

Deleting empty record from other nodes

Special control for deleting records

Amitay Isaacs CTDB database vacuuming for geniuses

slide-105
SLIDE 105

Existing issues with vacuuming

Vacuuming done in a child process

Forks a process from main ctdb daemon Can be very expensive

Migrating deleted record to lmaster

CTDB daemon itself cannot start record migration Migration is triggered from a client Hook vacuuming migration in recovery daemon (ugh!)

Deleting empty record from other nodes

Special control for deleting records Out-of-band modification to database

Amitay Isaacs CTDB database vacuuming for geniuses

slide-106
SLIDE 106

Existing issues with vacuuming

Vacuuming done in a child process

Forks a process from main ctdb daemon Can be very expensive

Migrating deleted record to lmaster

CTDB daemon itself cannot start record migration Migration is triggered from a client Hook vacuuming migration in recovery daemon (ugh!)

Deleting empty record from other nodes

Special control for deleting records Out-of-band modification to database Potential race with recovery

Amitay Isaacs CTDB database vacuuming for geniuses

slide-107
SLIDE 107

Existing issues with vacuuming

Vacuuming done in a child process

Forks a process from main ctdb daemon Can be very expensive

Migrating deleted record to lmaster

CTDB daemon itself cannot start record migration Migration is triggered from a client Hook vacuuming migration in recovery daemon (ugh!)

Deleting empty record from other nodes

Special control for deleting records Out-of-band modification to database Potential race with recovery

Hopefully, re-structuring will solve these problems!

Amitay Isaacs CTDB database vacuuming for geniuses

slide-108
SLIDE 108

CTDB restructure

Continuing from Martin’s talk . . .

Amitay Isaacs CTDB database vacuuming for geniuses

slide-109
SLIDE 109

CTDB restructure

Continuing from Martin’s talk . . . Database daemon Vacuuming daemon

Amitay Isaacs CTDB database vacuuming for geniuses

slide-110
SLIDE 110

CTDB restructure

Continuing from Martin’s talk . . . Database daemon

Focus on database operations

Vacuuming daemon

Amitay Isaacs CTDB database vacuuming for geniuses

slide-111
SLIDE 111

CTDB restructure

Continuing from Martin’s talk . . . Database daemon

Focus on database operations Single daemon – Most operations are done in child processes

Vacuuming daemon

Amitay Isaacs CTDB database vacuuming for geniuses

slide-112
SLIDE 112

CTDB restructure

Continuing from Martin’s talk . . . Database daemon

Focus on database operations Single daemon – Most operations are done in child processes

Vacuuming daemon

Light-weight process

Amitay Isaacs CTDB database vacuuming for geniuses

slide-113
SLIDE 113

CTDB restructure

Continuing from Martin’s talk . . . Database daemon

Focus on database operations Single daemon – Most operations are done in child processes

Vacuuming daemon

Light-weight process Keep track of deleted records

Amitay Isaacs CTDB database vacuuming for geniuses

slide-114
SLIDE 114

CTDB restructure

Continuing from Martin’s talk . . . Database daemon

Focus on database operations Single daemon – Most operations are done in child processes

Vacuuming daemon

Light-weight process Keep track of deleted records Handle failures better (what if records cannot deleted?)

Amitay Isaacs CTDB database vacuuming for geniuses

slide-115
SLIDE 115

CTDB restructure

Continuing from Martin’s talk . . . Database daemon

Focus on database operations Single daemon – Most operations are done in child processes

Vacuuming daemon

Light-weight process Keep track of deleted records Handle failures better (what if records cannot deleted?)

Hope that someone will write the code

Amitay Isaacs CTDB database vacuuming for geniuses

slide-116
SLIDE 116

Questions / Comments

Amitay Isaacs CTDB database vacuuming for geniuses