Cr Cruise Co Control: Effo l: Effortle less M Manage gement o - - PowerPoint PPT Presentation

▶
cr cruise co control effo l effortle less m manage gement
SMART_READER_LITE
LIVE PREVIEW

Cr Cruise Co Control: Effo l: Effortle less M Manage gement o - - PowerPoint PPT Presentation

Nov 25, 2022 478 likes •990 views

Cr Cruise Co Control: Effo l: Effortle less M Manage gement o of K f Kafka fka Clu Clusters Adem Efe Gencer Senior Software Engineer LinkedIn Kafka: A Distributed Stream Processing Platform : High throughput & low latency :

slide-1
SLIDE 1

Cr Cruise Co Control: Effo l: Effortle less M Manage gement o

  • f K

f Kafka fka Clu Clusters

Adem Efe Gencer

Senior Software Engineer LinkedIn

slide-2
SLIDE 2

Kafka: A Distributed Stream Processing Platform

: High throughput & low latency : Message persistence on partitioned data : Total ordering within each partition

2

slide-3
SLIDE 3

Key Concepts: Brokers, Topics, Partitions, and Replicas

: Broker-0 1 2 1 1 2 1 : Broker-1 : Broker-2

: A Replica of Partition-1 of Blue Topic

1 Kafka Cluster

3

slide-4
SLIDE 4

Key Concepts: Leaders and Followers

: Broker-0

: The Leader Replica

1 2 1 1 2 1 : Broker-1 : Broker-2 1

: A Follower Replica

1

4

slide-5
SLIDE 5

Key Concepts: Producers

Producer-1 Producer-2

: Broker-0 1 2 1 1 2 1 : Broker-1 : Broker-2

5

slide-6
SLIDE 6

Key Concepts: Consumers

Consumer-1 Consumer-2

: Broker-0 1 2 1 1 2 1 : Broker-1 : Broker-2

6

slide-7
SLIDE 7

Key Concepts: Failover via Leadership Transfer

: Broker-0 2 : Broker-2

✗

1 2 1 1 1

7

slide-8
SLIDE 8

Key Concepts: Failover via Leadership Transfer

: Broker-0 1 2 1 1 2 1 : Broker-2

✗

8

slide-9
SLIDE 9

Kafka Incurs Management Overhead

: Large deployments – e.g. @ : 2.6K+ Brokers, 44K+ Topics, 5M Partitions, 5T Messages / day : Frequent hardware failures : Load skew among brokers : Kafka cluster expansion and reduction

✗

“Elephant” (CC0): https://pixabay.com/en/elephant-safari-animal-defence-1421167, "Seesaw at Evelle" by Rachel Coleman (CC BY-SA 2.0): https://www.flickr.com/photos/rmc28/4862153119, “Inflatable Balloons” (Public Domain): https://commons.wikimedia.org/wiki/File:InflatableBalloons.jpg

9

slide-10
SLIDE 10

Alleviating the Management Overhead

1 2 3

Admin Operations for Cluster Maintenance Real-Time Monitoring of Kafka Clusters Anomaly Detection with Self-Healing

10

slide-11
SLIDE 11

Admin Operations for Cluster Maintenance

1

: Dynamically balance the cluster load : Add / remove brokers : Demote brokers – i.e. remove leadership of all replicas

+

  • : Trigger preferred leader election

: Fix offline replicas

11

slide-12
SLIDE 12

Admin Operations for Cluster Maintenance

1

: Dynamically balance the cluster load : Add / remove brokers : Demote brokers – i.e. remove leadership of all replicas

+

  • : Trigger preferred leader election

: Fix offline replicas

12

slide-13
SLIDE 13

Dynamically Balance the Cluster Load

: Never exceed the capacity of broker resources – e.g. disk, CPU, network bandwidth : Guarantee rack-aware distribution of replicas Must satisfy hard goals, including: : Enforce operational requirements – e.g. maximum replica count per broker

13

slide-14
SLIDE 14

Satisfy soft goals as much as possible – i.e. best effort : Balance replica distribution : Balance potential outbound network load : Balance distribution of partitions from the same topic : Balance disk, CPU, inbound/outbound network traffic utilization of brokers

✗ ✗

Dynamically Balance the Cluster Load

14

slide-15
SLIDE 15

Anomaly Detection with Self-Healing

2

: Goal violation – rebalance cluster : Broker failure – decommission broker(s) : Metric anomaly – demote broker(s)

✗

15

slide-16
SLIDE 16

Real-Time Monitoring of Kafka Clusters

3

: Check the health of brokers, disks, and user tasks : Examine the replica, leader, and load distribution : Identify under-replicated, under-min-ISR, and offline partitions

16

slide-17
SLIDE 17

Building Blocks of Management: Moving Replicas

: Broker-0 1 2 1 2 1 : Broker-1

Replica Move

17

slide-18
SLIDE 18

Building Blocks of Management: Moving Replicas

: Broker-0 1 2 1 2 : Broker-1

Replica Move

1

* Replica swap: Bidirectional reassignments of distinct partition replicas among brokers

Broader impact, but expensive

  • Requires data transfer*

18

slide-19
SLIDE 19

Building Blocks of Management: Moving Leadership

: Broker-0 1 2 1 2 1 : Broker-1

Leadership Move

19

slide-20
SLIDE 20

Building Blocks of Management: Moving Leadership

: Broker-0 1 2 1 2 1 : Broker-1

Leadership Move

Cheap, but has limited impact

  • Affects network bytes out and CPU

20

slide-21
SLIDE 21

A Multi-Objective Optimization Problem

Achieve conflicting cluster management goals while minimizing the impact of required

  • perations on user traffic

21

slide-22
SLIDE 22

ARCHITECTURE

“Joy Oil Gas Station Blueprints” (Public Domain): https://commons.wikimedia.org/wiki/File:Joy_Oil_gas_station_blueprints.jpg

slide-23
SLIDE 23

Cruise Control Architecture

REST API Monitor Executor Kafka Cluster

Metrics Reporter Sample Store Metric Sampler Goal(s)

Analyzer

Throttled Proposal Execution Reported Metrics Backup and Recovery Metrics Reporter T. Load History T. Broker Failures

Pluggable Component Anomaly Detector

Anomaly Notifier Goal Violation Metric Anomaly Broker Failure Finder(s)

Internal Topic

Capacity Resolver

  • Implements a public interface
  • Accepts custom user code
  • Created and used by Cruise

Control and its metrics reporter

23

slide-24
SLIDE 24

Metrics Reporter

Kafka Cluster

Metrics Reporter Metrics Reporter T. Load History T.

Produces selected Kafka cluster metrics to the configured metrics reporter topic with the configured frequency

24

slide-25
SLIDE 25

Monitor

Monitor

Sample Store Metric Sampler Capacity Resolver

Generates a model ( ) to describe the cluster

25

slide-26
SLIDE 26

Monitor: Cluster Model ( )

…

Monitoring windows disk cpu nw-in nw-out

time

latest utilization

: Load – current and historical utilization of brokers and replicas : Topology – rack, host, and broker distribution : Placement – replica, leadership, and partition distribution

26

slide-27
SLIDE 27

Monitor: Metric Sampler

Monitor Kafka Cluster

Sample Store Metric Sampler Reported Metrics Metrics Reporter T. Capacity Resolver

  • Periodically (e.g. every 5 min) consumes the reported

metrics to model the load on brokers and partitions

27

slide-28
SLIDE 28

Monitor: Sample Store

Monitor Kafka Cluster

Sample Store Metric Sampler Capacity Resolver

  • Produces broker and partition models to load history

topic, and uses the stored data to recover upon failure

Backup and Recovery Load History T.

28

slide-29
SLIDE 29

Monitor: Capacity Resolver

Monitor

Sample Store Metric Sampler Capacity Resolver

  • Gathers the broker capacities from a pluggable resolver

29

slide-30
SLIDE 30

Analyzer

Goal(s)

Analyzer

Generates proposals to achieve goals via a fast and near-optimal heuristic solution

30

slide-31
SLIDE 31

Analyzer: Goals

Goal(s)

Analyzer

Generates proposals to achieve goals via a fast and near-optimal heuristic solution

: Priorities – custom order of optimization : Strictness – hard (e.g. rack awareness) or soft (e.g. resource utilization balance) optimization demands : Modes – e.g. kafka-assigner (https://github.com/linkedin/kafka-tools)

31

slide-32
SLIDE 32

Analyzer: Proposals

Goal(s)

Analyzer

Generates proposals to achieve goals via a fast and near-optimal heuristic solution

+ =

Goal(s)

Proposals – in order of priority:

  • Leadership move > Replica move > Replica swap

32

slide-33
SLIDE 33

Executor

Executor Kafka Cluster

Throttled Proposal Execution

Proposal execution:

  • Dynamically controls the maximum number of

concurrent leadership / replica reassignments

  • Ensures only one execution at a time
  • Enables graceful cancellation of ongoing executions

Integration with replication quotas (KIP-73)

33

slide-34
SLIDE 34

Anomaly Detector

Anomaly Detector

Anomaly Notifier Goal Violation Metric Anomaly Broker Failure Finder(s)

Identifies, notifies, and fixes (self-healing):

  • Violation of anomaly detection goals
  • Broker failures
  • Metric anomalies

Disk failures (JBOD)

34

: Faulty vs Healthy Cluster : Reactive vs. Proactive Mitigation

slide-35
SLIDE 35

Anomaly Detector: Goal Violations and Self-Healing

Anomaly Detector

Anomaly Notifier Goal Violation Metric Anomaly Broker Failure Finder(s)

Checks for the violation of the anomaly detection goals

  • Identifies fixable and unfixable goal violations
  • Self-healing triggers a cluster rebalance operation
  • Avoids false positives due to broker failure, upgrade,

restart, or release certification

35

Healthy Faulty Proactive Reactive

slide-36
SLIDE 36

Anomaly Detector: Broker Failures

Anomaly Detector

Anomaly Notifier Goal Violation Metric Anomaly Broker Failure Finder(s)

Kafka Cluster

Broker Failures

Concerned with whether brokers are responsive:

  • Ignores the internal state deterioration of brokers
  • Identifies fail-stop failures

36

Healthy Faulty Proactive Reactive

slide-37
SLIDE 37

Anomaly Detector: Broker Failures and Self-Healing

Anomaly Detector

Anomaly Notifier Goal Violation Metric Anomaly Broker Failure Finder(s)

Kafka Cluster

Broker Failures

Checks for broker failures:

  • Enables a grace period to lower false positives – e.g.

due to upgrade, restart, or release certification

  • Self-healing triggers a remove operation for failed

brokers

37

Healthy Faulty Proactive Reactive

slide-38
SLIDE 38

Anomaly Detector: Reactive Mitigation

Anomaly Detector

Anomaly Notifier Goal Violation Metric Anomaly Broker Failure Finder(s)

Requires immediate attention of affected services Poor user experience due to frequent service interruptions Cluster maintenance becomes costly

~

Server & network failures

Size of clusters Volume of user traffic Hardware degradation

38

slide-39
SLIDE 39

Anomaly Detector: Metric Anomaly

Anomaly Detector

Anomaly Notifier Goal Violation Metric Anomaly Broker Failure Finder(s)

Checks for abnormal changes in broker metrics – e.g. a recent spike in log flush time:

  • Self-healing triggers a demote operation for slow

brokers

39

Healthy Faulty Proactive Reactive

slide-40
SLIDE 40

Anomaly Detector: Metric Anomaly

Anomaly Detector

Anomaly Notifier Goal Violation Metric Anomaly Broker Failure Finder(s)

Compares current and historical metrics to detect slow brokers:

  • The comparison in the default finder is based on the

percentile rank of the latest metric value

  • Metrics of interest are configurable – e.g. local time of

produce / consume / follower fetch, log flush time

  • Supports multiple active finders

40

Healthy Faulty Proactive Reactive

slide-41
SLIDE 41

Anomaly Detector: Proactive Mitigation

Anomaly Detector

Anomaly Notifier Goal Violation Metric Anomaly Broker Failure Finder(s)

In-place fix of slow / faulty brokers is non-trivial

  • The root cause could be a hardware issue (e.g. a

misbehaving disk), a software glitch, or a traffic shift

  • Hence, the mitigation strategies are agnostic of the

particular issue with the broker

41

“Three-toed-sloth” (CC BY 2.5): https://en.wikipedia.org/wiki/File:MC_Drei-Finger-Faultier.jpg

slide-42
SLIDE 42

REST API

REST API

Supports sync and async endpoints including: GUI & multi-cluster management

  • Cluster Load
  • Partition Load
  • Proposals
  • Kafka Cluster State
  • Cruise Control State
  • User Tasks

GET POST

  • Add / Remove / Demote Broker
  • Rebalance Cluster
  • Fix Offline Replicas (JBOD)
  • Stop Ongoing Execution
  • Pause / Resume Sampling
  • Admin – ongoing behavior changes

42

slide-43
SLIDE 43

Managing the Manager – Monitoring Cruise Control

Reported JMX metrics include: : Broker failure, goal violation, and metric anomaly rate : Cluster model and sampling performance : Stats on proposal generation : Started, stopped, and ongoing executions in different modes, and the status of balancing tasks

Executor Anomaly Detector Monitor Analyzer

43

slide-44
SLIDE 44

Evaluation: Remove Brokers and Rebalance

2 1 All Topics Bytes-In Rate [per broker]

Incoming data (bytes/s) Time (hours)

44

slide-45
SLIDE 45

Evaluation: Remove Brokers and Rebalance

2 1 All Topics Bytes-Out Rate [per broker]

Outgoing data (bytes/s) Time (hours)

45

slide-46
SLIDE 46

Evaluation: Remove Brokers and Rebalance

2 1 Number of Partitions [per broker]

Partition count Time (hours)

46

slide-47
SLIDE 47

Summary

A system that provides effortless management of Kafka clusters

Admin Operations for Cluster Maintenance Anomaly Detection with Self-healing Integration with Other Systems – e.g. Apache Helix Real-Time Monitoring of Kafka Clusters

47

slide-48
SLIDE 48

More…

: Open source repository (https://github.com/linkedin/cruise-control) : Gitter room (https://gitter.im/kafka-cruise-control)

48

: UI (https://github.com/linkedin/cruise-control-ui)

slide-49
SLIDE 49

Rate today ’s session

Session page on conference website O’Reilly Events App

slide-50
SLIDE 50