Nomad HASHICORP Armon Dadgar @armon HASHICORP HASHICORP Cluster - - PowerPoint PPT Presentation

▶
nomad
SMART_READER_LITE
LIVE PREVIEW

Nomad HASHICORP Armon Dadgar @armon HASHICORP HASHICORP Cluster - - PowerPoint PPT Presentation

Mar 28, 2024 1.59k likes •2.4k views

Nomad HASHICORP Armon Dadgar @armon HASHICORP HASHICORP Cluster Manager Scheduler Nomad HASHICORP Cluster Manager Scheduler Nomad HASHICORP Schedulers map a set of work to a set of resources HASHICORP Work (Input) Resources Web

slide-1
SLIDE 1

Nomad

HASHICORP

slide-2
SLIDE 2

HASHICORP

Armon Dadgar

@armon

slide-3
SLIDE 3
slide-4
SLIDE 4

HASHICORP

slide-5
SLIDE 5

Nomad

HASHICORP

Cluster Manager Scheduler

slide-6
SLIDE 6

Nomad

HASHICORP

Cluster Manager Scheduler

slide-7
SLIDE 7

HASHICORP

Schedulers map a set of work to a set of resources

slide-8
SLIDE 8

HASHICORP

CPU Scheduler

Web Server -Thread 1 CPU - Core 1 CPU - Core 2 Web Server -Thread 2 Redis -Thread 1 Kernel -Thread 1

Work (Input) Resources

CPU Scheduler

slide-9
SLIDE 9

HASHICORP

CPU Scheduler

Web Server -Thread 1 CPU - Core 1 CPU - Core 2 Web Server -Thread 2 Redis -Thread 1 Kernel -Thread 1

Work (Input) Resources

CPU Scheduler

slide-10
SLIDE 10

HASHICORP

Schedulers In the Wild

Type Work Resources CPU Scheduler Threads Physical Cores AWS EC2 / OpenStack Nova Virtual Machines Hypervisors Hadoop YARN MapReduce Jobs Client Nodes Cluster Scheduler Applications Servers

slide-11
SLIDE 11

HASHICORP

Advantages

Higher Resource Utilization Decouple Work from Resources Better Quality of Service

slide-12
SLIDE 12

HASHICORP

Advantages

Bin Packing Over-Subscription Job Queueing Higher Resource Utilization Decouple Work from Resources Better Quality of Service

slide-13
SLIDE 13

HASHICORP

Advantages

Abstraction API Contracts Standardization Higher Resource Utilization Decouple Work from Resources Better Quality of Service

slide-14
SLIDE 14

HASHICORP

Advantages

Priorities Resource Isolation Pre-emption Higher Resource Utilization Decouple Work from Resources Better Quality of Service

slide-15
SLIDE 15

HASHICORP

slide-16
SLIDE 16

Nomad

HASHICORP

slide-17
SLIDE 17

Nomad

HASHICORP

Cluster Scheduler Easily Deploy Applications Operationally Simple Built for Scale

slide-18
SLIDE 18

job "redis" { datacenters = ["us-east-1"] task "redis" { driver = "docker" config { image = "redis:latest" } resources { cpu = 500 # Mhz memory = 256 # MB network { mbits = 10 dynamic_ports = ["redis"] } } } }

example.nomad

slide-19
SLIDE 19

HASHICORP

Job Specification

Declares what to run

slide-20
SLIDE 20

HASHICORP

Job Specification

Nomad determines where and manages how to run

slide-21
SLIDE 21

HASHICORP

Job Specification

Nomad abstracts work from resources

slide-22
SLIDE 22

HASHICORP

Containerized Virtualized Standalone

Docker Rkt Windows Server Containers Qemu / KVM Hyper-V Xen Java Jar Static Binaries C#

slide-23
SLIDE 23

Nomad

HASHICORP

Declarative Job Specification Infrastructure-As-Code Removes Imperative Logic External Dependencies?

slide-24
SLIDE 24

Nomad

HASHICORP

Service Discovery? Health Monitoring? Application Secrets? Stateful Applications?

slide-25
SLIDE 25

job “my-app" { … task “my-app" { service { port = “http” check { type = “http” path = “/health” interval = “5s” } } } }

example.nomad

slide-26
SLIDE 26

HASHICORP

Nomad Server Consul Server Client Nomad Consul App 1 App N

Schedule App Register Service Monitor Health

slide-27
SLIDE 27

Nomad

HASHICORP

Secret Distribution: API Keys DB Credentials SSL/TLS Certificates

slide-28
SLIDE 28

job “my-app" { … task “my-app" { env { DB_USERPASS = “foo:bar” } } }

example.nomad

slide-29
SLIDE 29

Vault

Secure secret storage Dynamic secrets Leasing, renewal, and revocation Auditing Rich ACLs Multiple client authentication methods

slide-30
SLIDE 30

HASHICORP

Login Vault Token

Vault Token + Operation Op Response

slide-31
SLIDE 31

job “my-app" { … task “my-app" { env { VAULT_TOKEN = “b6a10b96-9060-11e6-9c6f-67a52bc6b8d3” } } }

example.nomad

slide-32
SLIDE 32

job “my-app" { … task “my-app" { vault { policies = [“my-app-role”] } } }

example.nomad

slide-33
SLIDE 33

HASHICORP

Nomad Server Client Nomad App 1 App N

Submit Job + Vault Token Verify Vault Token Schedule App Generate + Renew Vault Token

slide-34
SLIDE 34

Nomad

HASHICORP

Native Vault Integration No Secrets in Jobs No Secrets on Client Disk Minimize Trust

slide-35
SLIDE 35

HASHICORP

Stateful Applications

Stateless Stateful

slide-36
SLIDE 36

HASHICORP

Stateful Applications

Stateless Stateful API Web Cache

slide-37
SLIDE 37

HASHICORP

Stateful Applications

Stateless Stateful API Web Cache HDFS Cassandra MongoDB

slide-38
SLIDE 38

HASHICORP

Stateful Applications

Stateless Stateful API Web Cache HDFS Cassandra MongoDB *SQL

slide-39
SLIDE 39

HASHICORP

Stateful Applications

Stateless Stateful API Web Cache HDFS Cassandra MongoDB *SQL EASY MEDIUM HARD

slide-40
SLIDE 40

job “my-app" { … task “my-app" { ephemeral_disk { sticky = true } } }

example.nomad

slide-41
SLIDE 41

HASHICORP

Moves data between tasks on the same machine

slide-42
SLIDE 42

HASHICORP

Copies data between tasks on different machines

slide-43
SLIDE 43

Nomad

HASHICORP

Easily Deploy Apps: Declarative Jobs Flexible Workloads Consul Integration Vault Integration Sticky Volumes

slide-44
SLIDE 44

HASHICORP

Operationally Simple

slide-45
SLIDE 45

HASHICORP

Client Server

slide-46
SLIDE 46

Built on Experience

GOSSIP CONSENSUS

slide-47
SLIDE 47

Serf

Cluster Management Gossip Based (P2P) Membership Failure Detection Event System

slide-48
SLIDE 48

Serf

Large Scale Production Hardened Simple Clustering and Federation

slide-49
SLIDE 49

Consul

Service Discovery Configuration Coordination (Locking) Central Servers + Distributed Clients

slide-50
SLIDE 50

Consul

Multi-Datacenter Raft Consensus Large Scale Production Hardened

slide-51
SLIDE 51

Nomad

HASHICORP

Operational Simplicity: Single Binary No Dependencies Highly Available

slide-52
SLIDE 52

HASHICORP

Built for Scale

slide-53
SLIDE 53

Built on Experience

GOSSIP CONSENSUS

Mature Libraries Proven Design Patterns Lacking Scheduling Logic

slide-54
SLIDE 54

Built on Research

GOSSIP CONSENSUS

slide-55
SLIDE 55
slide-56
SLIDE 56

HASHICORP

Single Region Architecture

SERVER SERVER SERVER CLIENT CLIENT CLIENT

DC1 DC2 DC3 FOLLOWER LEADER FOLLOWER REPLICATION FORWARDING REPLICATION FORWARDING RPC RPC RPC

slide-57
SLIDE 57

HASHICORP

Multi Region Architecture

SERVER SERVER SERVER

FOLLOWER LEADER FOLLOWER REPLICATION FORWARDING REPLICATION

REGION B

GOSSIP REPLICATION REPLICATION FORWARDING REGION FORWARDING

 REGION A

SERVER

FOLLOWER

SERVER SERVER

LEADER FOLLOWER

slide-58
SLIDE 58

Nomad

HASHICORP

Region is Isolation Domain 1-N Datacenters Per Region Flexibility to do 1:1 (Consul) Scheduling Boundary

slide-59
SLIDE 59

HASHICORP

Hundreds of regions Tens of thousands of clients per region Thousands of jobs per region

slide-60
SLIDE 60

Nomad

Inspired by Google Omega Optimistic Concurrency State Coordination Service & Batch workloads Pluggable Architecture

slide-61
SLIDE 61

Data Model

ALLOCATION JOB EVALUATION NODE

slide-62
SLIDE 62

Evaluation ~= State Change

slide-63
SLIDE 63

Evaluations

Create / Update / Delete Job Node Up / Node Down Allocation Failed / Finished

slide-64
SLIDE 64

Evaluations

SCHEDULER

func(Evaluation) => []AllocationUpdates

slide-65
SLIDE 65

Evaluations

SCHEDULER

func(Evaluation) => []AllocationUpdates

Service, Batch, System

slide-66
SLIDE 66

HASHICORP

External Event EvaluaBon CreaBon EvaluaBon Queuing EvaluaBon Processing OpBmisBc CoordinaBon State Updates

slide-67
SLIDE 67

Nomad

HASHICORP

Omega Architecture Optimistically Schedule 100’s of Jobs in Parallel Controls for Correctness

slide-68
SLIDE 68

Nomad

Million Container Challenge

1,000 Jobs 1,000 Tasks per Job 5,000 Hosts on GCE 1,000,000 Containers

slide-69
SLIDE 69
slide-70
SLIDE 70

“

– Bill Gates

640 KB ought to be enough for anybody.

slide-71
SLIDE 71

2nd Largest Hedge Fund 18K Cores 5 Hours 2,200 Containers/second

slide-72
SLIDE 72

Nomad

HASHICORP

Cluster Scheduler Easily Deploy Applications Operationally Simple Built for Scale

slide-73
SLIDE 73

HASHICORP

Thanks!

Q/A

slide-74
SLIDE 74

HASHICORP