MesosCon Asia 2016 Getting Rid of Zookeeper Jay Guo Kapil Arya Software Developer @IBM Mesos Committer @Mesosphere guojiannan@cn.ibm.com kapil@mesosphere.io 1
Motivation 2
Zookeeper is... Mature ● Feature-rich ● ... ● 3
But! Primitive K/V store ● Provide your own tooling for other abstractions! ○ Heavy ● Hard dependencies ● Language binding instead of RESTfull API ● ... ● 4
It’s all about having options! Chocolate ● Strawberry ● Vanilla ● ... ● 5
Mesos HA: An Overview 6
High Availability First of all, we need a Distributed Key-Value storage... 7
Mesos HA Mesos leader ● At least three Mesos Masters Master ● One leading Master ○ Leader election Mesos Agents / Frameworks ○ Leader detection Zookeeper ZK ZK cluster ZK ● Replicated Log Mesos Mesos Master Master Zookeeper as the distributed key-value store 8
Mesos HA: Leader Election Mesos All Masters “contend” to be ● Master the leader! Contend ZK ZK ZK Contend Contend Mesos Mesos Master Master 9
Mesos HA: Leader Election Mesos All Masters “contend” to be ● Master the leader! Fail Only one succeeds; others fail ● ZK ZK ZK Success Fail Mesos Mesos Master Master 10
Mesos HA: Leader Election Mesos All Masters “contend” to be ● Master the leader! Watch Only one succeeds; others fail ● We have a leading Masters! ● ZK ZK ZK Hold Watch Mesos Mesos Master Master 11
Mesos HA: Losing a Leader Mesos Suppose the leading Master is ● Master “ lost ” Watch ZK ZK ZK Master connection Watch lost Mesos Mesos Master Master 12
Mesos HA: Losing a Leader Mesos Suppose the leading Master is ● Master “ lost ” Notify All other Masters are notified ● ZK ZK ZK Master connection Notify lost Mesos Mesos Master Master 13
Mesos HA: Losing a Leader Mesos Suppose the leading Master is ● Master “ lost ” Contend All other Masters are notified ● The remaining Masters ● contend again ZK ZK ZK Contend Mesos Mesos Master Master 14
Mesos HA: Losing a Leader Mesos Suppose the leading Master is ● Master “ lost ” Success All other Masters are notified ● The remaining Masters ● contend again ZK ZK ZK One of them succeeds ● Fail Mesos Mesos Master Master 15
Mesos HA: Losing a Leader Mesos Suppose the leading Master is ● Master “ lost ” Watch All other Masters are notified ● The remaining Masters ● contend again ZK ZK ZK One of them succeeds ● Hold Mesos Mesos Master Master A new leader is elected! ● 16
What about Agents/Frameworks? 17
Mesos HA: Leader Detection Mesos Framework/Agent connects to ● Master Zookeeper to “ detect ” about Mesos Agents Watch / Frameworks the current leading Master Detect ZK ZK ZK Hold Watch Mesos Mesos Master Master 18
Mesos HA: Leader Detection Mesos Framework/Agent connects to ● Master Zookeeper to “ detect ” about Mesos Agents Watch / Frameworks the current leading Master Zookeeper provides Master’s ● IP:Port location ZK ZK ZK I.e. IP:Port ○ Hold Watch Mesos Mesos Master Master 19
Mesos HA: Leader Detection Mesos Framework/Agent connects to ● Master Zookeeper to “ detect ” the Mesos Agents Watch / Frameworks current leading Master Zookeeper provides Master’s ZK ● Connect ZK ZK location Hold Watch Framework/Agent connects ● Mesos Mesos Master Master to the “ leader ” 20
What about Replicated Log? Replicated Log lets you create replicated fault-tolerant append-only logs. The Mesos master uses Replicated Log to store cluster state in a replicated, durable way. 21
Mesos HA: Replicated Log Replica Each replica registers its pid ● into ZK and maintain the Register & hold presence. ZK ZK ZK Register & hold Replica 22
Mesos HA: Replicated Log Replica Each replica registers its pid ● into ZK and maintain the Notified with info presence. of new replica When new replica joins the ● cluster, existing ones get ZK ZK ZK notified and get to know the Notified with info register of new replica pid of new replica. Replica Replica 23
Mesos HA: Replicated Log Replica Each replica registers its own ● pid into ZK and maintain the presence. Paxos Paxos When new replica joins the ● cluster, existing ones get ZK ZK ZK notified and get to know the pid of new replica. Every replica knows all nodes ● Replica Replica in the cluster and do Paxos . Paxos 24
Replacing leader Mesos Master Zookeeper Distributed ? KV Store ZK ZK Mesos Mesos Master Master ? = Etcd || Consul || ZK ... 25
Three Key Components Master Contender for leader election ● Mesos Master bool contend (); Contender ? Distributed ZK ZK KV Store 26
Three Key Components Master Contender for leader election ● Mesos Master bool contend (); Detector Contender Master Detector for discovery ● MasterInfo detect (MasterInfo previous); ? Distributed ZK ZK KV Store 27
Three Key Components Master Contender for leader election ● Mesos Master bool contend (); Detector Contender PIDGroup Master Detector for discovery ● MasterInfo detect (MasterInfo previous); PIDGroup for initialization ● ? Distributed ZK void initialize (pid_t pid); ZK KV Store 28
A Case for Modularization! Mesos Already a clear-cut interfaces between: ● Master Master and Contender ○ Agent and Detector ○ Detector Contender PIDGroup Framework and Detector ○ For new distributed KV store ● implementation, we just write the module without having to modify Mesos ? Distributed ZK ZK KV Store itself! 29
Let’s Talk about Modules! 30
Mesos ● Module/Plugin/Extension ● Add/replace a Mesos component Modules Isolators ○ Authenticators ○ … ○ ● Hook modules: Listen to interesting events ○ Modify/enhance certain code paths ○ Prepare/enhance task environment ○ ... ○ 31
How are Modules Used? Compiled as shared libraries ● ○ E.g., libmesos_network_overlay.so Specified when launching Master/Agent/Framework ● mesos-agent.sh <master-parameters> --modules=file:///path/to/modules.json --isolation=”my_isolator” Gets loaded during initialization ● E.g., the ”my_isolator” isolator will be loaded into the Agent to ○ provide task isolation 32
Community Modules I just wrote a Mesos module that provides a really cute feature. How do I make it useful for others! 33
Modules are Tricky! Developing ● Building ● Testing ● Using ● Hosting ● How can we make it all better for community? ● 34
Writing Modules Doesn’t require intimate Mesos knowledge ● Just the details of the subsystem being implemented (e.g., Isolators) ○ Familiarity with Mesos model is required ● E.g., libprocess, events, futures and promises, etc. ○ Closely tied with Mesos version ● To ensure mutual compatibility ○ 35
Building Modules: Issues Build Mesos first! ● Install all Mesos dependencies ○ Takes a long time to build ○ Version dependencies ○ 36
Building Modules: Good News! Starting Mesos 1.0 release, pre-compiled Mesos deb/rpm packages ● contain everything needed to build modules 37
Testing Modules I just wrote a simple Mesos module that provide a cute feature and I know how to build it! Can I write unit tests for it? 38
Testing Modules Key questions: ● How to get good test coverage? ○ How can we solicit help from community? ○ Good news! ● Efforts on the way to create a “libmesos_test” library that can be used to ○ create/run gmock style tests just like with Mesos itself. 39
Community-Driven Modules How do we, as a community, make third-party modules available for general consumption? While making sure the developers and consumers can seamlessly test/integrate into their environments! 40
Community Modules: Proposal A central registry that contains pointers: ● E.g., github.com/mesos/modules ○ Each module (or a set of related modules) in its own repository ○ Make Mesos version-specific binary rpm/deb modules available ● E.g., lib_my_module_< module-version >_< mesos_version >.so ○ 41
Module CI: Coming Soon! Builds binary packages for every registered module ● Across a given set of Mesos versions ○ Work-in-progress! ○ Automatic build/testing for upcoming Mesos release ● Catch incompatibilities sooner! ○ Run tests! ● 42
Let’s take a look at Etcd! 43
Etcd: HTTP API (no language bindings) ● May already exist in your environments ● A Distributed KV Store 44
Etcd in a Mesos Cluster Mesos Create Etcd-specific modules for: ● Master Master detector ○ Master Contender ○ Detector Contender PIDGroup PIDGroup ○ No need to modify/rebuild Mesos ● Distributed Etcd ZK ZK KV Store 45
Again, it’s all about having options! Chocolate ● Strawberry ● Vanilla ● ... ● 46
Again, it’s all about having options! Chocolate ● Zookeeper Strawberry ● Etcd Vanilla ● Consul ... ● 47
Recommend
More recommend
