The CTDB Report Martin Schwenke < martin@meltin.net > Amitay - - PowerPoint PPT Presentation

▶
the ctdb report
SMART_READER_LITE
LIVE PREVIEW

The CTDB Report Martin Schwenke < martin@meltin.net > Amitay - - PowerPoint PPT Presentation

Oct 04, 2022 359 likes •1.07k views

The CTDB Report Martin Schwenke < martin@meltin.net > Amitay Isaacs < amitay@samba.org > Samba Team IBM (Australia Development Laboratory, Linux Technology Center) SambaXP 2019 Martin Schwenke, Amitay Isaacs The CTDB Report

slide-1
SLIDE 1

The CTDB Report

Martin Schwenke <martin@meltin.net> Amitay Isaacs <amitay@samba.org>

Samba Team IBM (Australia Development Laboratory, Linux Technology Center)

SambaXP 2019

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-2
SLIDE 2

Overview

Progress in the past year Plans presented at SambaXP 2017/2018 Design ideas New daemons Way forward

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-3
SLIDE 3

Progress in the past year

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-4
SLIDE 4

Progress in the past year

Committers

Alexander Bokovoy 4 Amitay Isaacs 82 Andreas Schneider 11 Andrew Bartlett 3 Carlos O’Donell 1 Christof Schmitt 3 David Disseldorp 8 Douglas Bagnall 1 Martin Schwenke 315 Noel Power 4 Olly Betts 1 Rafael David Tinoco 1 Ralph Boehme 1 Ralph Wuerthner 1 Samuel Cabrero 1 Stefan Metzmacher 5 Swen Schillig 17 Volker Lendecke 6 Zhu Shangzhong 1 466

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-5
SLIDE 5

Progress in the past year

Commits by area

Configuration changes for 4.9 23 Add eventd (including preparation + fixes) 64 Portability 32 Portability - Packet handling 35 Recovery lock reliability 20 Vacuuming improvements 11 Scripts - NFS fixes for systemd 13 Test - local_daemons.sh 53 Test - generic improvements 52 Build/WAF 2.0/Py3 22 Generic Samba clean-ups 13 Other 128 466

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-6
SLIDE 6

Plans presented at SambaXP 2017/2018

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-7
SLIDE 7

Plans presented at SambaXP 2017/2018

Separate daemons event daemon service daemon failover daemon + connection tracking daemon cluster daemon database daemon transport smbd proxy

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-8
SLIDE 8

Plans presented at SambaXP 2017/2018

eventd serviced failoverd + conntrackd clusterd + databased transport

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-9
SLIDE 9

Plans presented at SambaXP 2017/2018

eventd In Samba 4.9 serviced failoverd + conntrackd clusterd + databased transport

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-10
SLIDE 10

Plans presented at SambaXP 2017/2018

eventd In Samba 4.9 serviced Initial version finished before 4.9 failoverd + conntrackd clusterd + databased transport

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-11
SLIDE 11

Plans presented at SambaXP 2017/2018

eventd In Samba 4.9 serviced Initial version finished before 4.9 failoverd + conntrackd Hurriedly, nearly finished before 4.9 A lot of copy & paste from serviced Could have gone into 4.9. . . . . . but required lots of integration work clusterd + databased transport

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-12
SLIDE 12

Plans presented at SambaXP 2017/2018

eventd In Samba 4.9 serviced Initial version finished before 4.9 failoverd + conntrackd Hurriedly, nearly finished before 4.9 A lot of copy & paste from serviced Could have gone into 4.9. . . . . . but required lots of integration work clusterd + databased Not implemented transport

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-13
SLIDE 13

Plans presented at SambaXP 2017/2018

eventd In Samba 4.9 serviced Initial version finished before 4.9 failoverd + conntrackd Hurriedly, nearly finished before 4.9 A lot of copy & paste from serviced Could have gone into 4.9. . . . . . but required lots of integration work clusterd + databased Not implemented transport In design phase

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-14
SLIDE 14

Plans presented at SambaXP 2017/2018

Status Conclusions

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-15
SLIDE 15

Plans presented at SambaXP 2017/2018

Status Components not mature enough for 4.9, not merged Conclusions

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-16
SLIDE 16

Plans presented at SambaXP 2017/2018

Status Components not mature enough for 4.9, not merged Conclusions Lots of boilerplate code for each daemon and client tool Each daemon with a unix domain socket Separate protocol for each daemon

Client — Server Server — Server ?

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-17
SLIDE 17

Plans presented at SambaXP 2017/2018

Status Components not mature enough for 4.9, not merged Conclusions Lots of boilerplate code for each daemon and client tool Each daemon with a unix domain socket Separate protocol for each daemon

Client — Server Server — Server ?

sock_daemon

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-18
SLIDE 18

Plans presented at SambaXP 2017/2018

Status Components not mature enough for 4.9, not merged Conclusions Lots of boilerplate code for each daemon and client tool Each daemon with a unix domain socket Separate protocol for each daemon

Client — Server Server — Server ?

sock_daemon Testing becomes easier

No need for fake daemons

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-19
SLIDE 19

Plans presented at SambaXP 2017/2018

Status Components not mature enough for 4.9, not merged Conclusions Lots of boilerplate code for each daemon and client tool Each daemon with a unix domain socket Separate protocol for each daemon

Client — Server Server — Server ?

sock_daemon Testing becomes easier

No need for fake daemons

. . . and complicated

serviced → eventd failoverd → eventd, transport Need multiple daemons for setup

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-20
SLIDE 20

Design ideas

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-21
SLIDE 21

Design ideas

Topics Reduce copy/paste code Simplify testing Unify protocol Too many sockets

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-22
SLIDE 22

Design ideas

Reduce copy/paste code

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-23
SLIDE 23

Design ideas

Reduce copy/paste code sock_daemon was good for abstracting Forces boilerplate code for each daemon Avoids handling protocol, but not very effective

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-24
SLIDE 24

Design ideas

Reduce copy/paste code sock_daemon was good for abstracting Forces boilerplate code for each daemon Avoids handling protocol, but not very effective Enter tdaemon

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-25
SLIDE 25

Design ideas

Reduce copy/paste code sock_daemon was good for abstracting Forces boilerplate code for each daemon Avoids handling protocol, but not very effective Enter tdaemon And possibly tclient

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-26
SLIDE 26

Design ideas

Simplify testing

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-27
SLIDE 27

Design ideas

Simplify testing Unit testing of ctdb daemon is impossible! Separate daemons are easier to unit test How to handle dependncies? Can we combine multiple daemons?

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-28
SLIDE 28

Design ideas

Simplify testing Unit testing of ctdb daemon is impossible! Separate daemons are easier to unit test How to handle dependncies? Can we combine multiple daemons? Each daemon is a tevent_req computation . . . One daemon to rule them all?

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-29
SLIDE 29

Design ideas

Simplify testing Unit testing of ctdb daemon is impossible! Separate daemons are easier to unit test How to handle dependncies? Can we combine multiple daemons? Each daemon is a tevent_req computation . . . One daemon to rule them all? masterd

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-30
SLIDE 30

Design ideas

Unify protocol

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-31
SLIDE 31

Design ideas

Unify protocol Each daemon needs some common “controls” Should Client — Server be different from Server — Server? New protocol? Design it right from beginning – endian neutral Common “controls” can be implemented once

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-32
SLIDE 32

Design ideas

Unify protocol Each daemon needs some common “controls” Should Client — Server be different from Server — Server? New protocol? Design it right from beginning – endian neutral Common “controls” can be implemented once tdaemon

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-33
SLIDE 33

Design ideas

Too many sockets

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-34
SLIDE 34

Design ideas

Too many sockets Each daemon with unix domain socket Easy to test, . . . . . . but gets messy to manage many sockets

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-35
SLIDE 35

Design ideas

Too many sockets Each daemon with unix domain socket Easy to test, . . . . . . but gets messy to manage many sockets Messaging server? . . . Unix databagram messaging

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-36
SLIDE 36

Design ideas

Too many sockets Each daemon with unix domain socket Easy to test, . . . . . . but gets messy to manage many sockets Messaging server? . . . Unix databagram messaging transportd Every daemon now uses common transport client code

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-37
SLIDE 37

Design ideas

Too many sockets Each daemon with unix domain socket Easy to test, . . . . . . but gets messy to manage many sockets Messaging server? . . . Unix databagram messaging transportd Every daemon now uses common transport client code Works very well for tdaemon abstraction

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-38
SLIDE 38

New daemons

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-39
SLIDE 39

New daemon

Topics Master daemon Transport daemon

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-40
SLIDE 40

New daemons

Master daemon

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-41
SLIDE 41

New daemons

Master daemon Start and monitor multiple daemons

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-42
SLIDE 42

New daemons

Master daemon Start and monitor multiple daemons

Multiple process model Single process model

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-43
SLIDE 43

New daemons

Master daemon Start and monitor multiple daemons

Multiple process model Single process model

Bundle all dependencies for testing in one daemon

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-44
SLIDE 44

New daemons

Master daemon Start and monitor multiple daemons

Multiple process model Single process model

Bundle all dependencies for testing in one daemon No, this is not systemd :-)

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-45
SLIDE 45

New daemons

Transport daemon

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-46
SLIDE 46

New daemons

Transport daemon All daemons talk to transport Routes packets between daemons Routes packets between nodes Understands just enough protocol for routing Keep it light and blazing fast!

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-47
SLIDE 47

New daemons

Transport daemon All daemons talk to transport Routes packets between daemons Routes packets between nodes Understands just enough protocol for routing Keep it light and blazing fast! Further ideas

Minimise dynamic memory allocation. . . . . . to zero?

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-48
SLIDE 48

Way forward

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-49
SLIDE 49

Way forward

Incremental development

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-50
SLIDE 50

Way forward

Incremental development To make best long-term progress, avoid churn

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-51
SLIDE 51

Way forward

Incremental development To make best long-term progress, avoid churn To avoid churn, we need to develop against transportd API

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-52
SLIDE 52

Way forward

Incremental development To make best long-term progress, avoid churn To avoid churn, we need to develop against transportd API Either need to develop new database daemon against transportd API. . .

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-53
SLIDE 53

Way forward

Incremental development To make best long-term progress, avoid churn To avoid churn, we need to develop against transportd API Either need to develop new database daemon against transportd API. . . . . . or retrofit existing ctdbd against transportd API

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-54
SLIDE 54

Way forward

Incremental development To make best long-term progress, avoid churn To avoid churn, we need to develop against transportd API Either need to develop new database daemon against transportd API. . . . . . or retrofit existing ctdbd against transportd API The latter involves significant churn

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-55
SLIDE 55

Way forward

Fake transportd client

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-56
SLIDE 56

Way forward

Fake transportd client Implement alternative transportd client code that uses current ctdbd as transport?

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-57
SLIDE 57

Way forward

Fake transportd client Implement alternative transportd client code that uses current ctdbd as transport? Implement new components using this API

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-58
SLIDE 58

Way forward

Fake transportd client Implement alternative transportd client code that uses current ctdbd as transport? Implement new components using this API Implement new database daemon and transportd

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-59
SLIDE 59

Way forward

Fake transportd client Implement alternative transportd client code that uses current ctdbd as transport? Implement new components using this API Implement new database daemon and transportd However, first step involves churn

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-60
SLIDE 60

Way forward

Recovery scalability

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-61
SLIDE 61

Way forward

Recovery scalability Recovery master node is (probably) a bottleneck for recovery

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-62
SLIDE 62

Way forward

Recovery scalability Recovery master node is (probably) a bottleneck for recovery Recovery master could distribute recovery of individual databases across nodes

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-63
SLIDE 63

Way forward

Recovery scalability Recovery master node is (probably) a bottleneck for recovery Recovery master could distribute recovery of individual databases across nodes Could implement in current code

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-64
SLIDE 64

Way forward

Recovery scalability Recovery master node is (probably) a bottleneck for recovery Recovery master could distribute recovery of individual databases across nodes Could implement in current code Churn!

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-65
SLIDE 65

Way forward

Problem Every time we churn we delay progress towards new design. . .

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-66
SLIDE 66

Way forward

CTDB developers needed Samba Team has one full time CTDB developer

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-67
SLIDE 67

Way forward

CTDB developers needed Samba Team has one full time CTDB developer Some amount of burnout. . .

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-68
SLIDE 68

Way forward

CTDB developers needed Samba Team has one full time CTDB developer Some amount of burnout. . . Any volunteers?

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-69
SLIDE 69

Legal Statement

This work represents the view of the authors and does not necessarily represent the view of IBM. IBM is a registered trademark of International Business Machines Corporation in the United States and/or other countries. Linux is a registered trademark of Linus Torvalds. Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both. Other company, product, and service names may be trademarks or service marks of others.

Martin Schwenke, Amitay Isaacs The CTDB Report

slide-70
SLIDE 70

Questions?

Martin Schwenke, Amitay Isaacs The CTDB Report