Nomad HASHICORP
Armon Dadgar @armon HASHICORP
HASHICORP
Cluster Manager Scheduler Nomad HASHICORP
Cluster Manager Scheduler Nomad HASHICORP
Schedulers map a set of work to a set of resources HASHICORP
Work (Input) Resources Web Server -Thread 1 CPU - Core 1 Web Server -Thread 2 CPU Scheduler Redis -Thread 1 CPU - Core 2 Kernel -Thread 1 HASHICORP CPU Scheduler
Work (Input) Resources Web Server -Thread 1 CPU - Core 1 Web Server -Thread 2 CPU Scheduler Redis -Thread 1 CPU - Core 2 Kernel -Thread 1 HASHICORP CPU Scheduler
Type Work Resources CPU Scheduler Threads Physical Cores AWS EC2 / OpenStack Nova Virtual Machines Hypervisors Hadoop YARN MapReduce Jobs Client Nodes Cluster Scheduler Applications Servers HASHICORP Schedulers In the Wild
Higher Resource Utilization Decouple Work from Resources Better Quality of Service HASHICORP Advantages
Higher Resource Utilization Bin Packing Decouple Work from Resources Over-Subscription Better Quality of Service Job Queueing HASHICORP Advantages
Higher Resource Utilization Abstraction Decouple Work from Resources API Contracts Better Quality of Service Standardization HASHICORP Advantages
Higher Resource Utilization Priorities Decouple Work from Resources Resource Isolation Better Quality of Service Pre-emption HASHICORP Advantages
HASHICORP
Nomad HASHICORP
Cluster Scheduler Easily Deploy Applications Operationally Simple Built for Scale Nomad HASHICORP
example.nomad job "redis" { datacenters = ["us-east-1"] task "redis" { driver = "docker" config { image = "redis:latest" } resources { cpu = 500 # Mhz memory = 256 # MB network { mbits = 10 dynamic_ports = ["redis"] } } } }
Declares what to run HASHICORP Job Specification
Nomad determines where and manages how to run HASHICORP Job Specification
Nomad abstracts work from resources HASHICORP Job Specification
Docker Containerized Rkt Windows Server Containers Qemu / KVM Virtualized Xen Hyper-V Java Jar Standalone Static Binaries C# HASHICORP
Declarative Job Specification Infrastructure-As-Code Removes Imperative Logic External Dependencies? Nomad HASHICORP
Service Discovery? Health Monitoring? Application Secrets? Stateful Applications? Nomad HASHICORP
example.nomad job “my-app" { … task “my-app" { service { port = “http” check { type = “http” path = “/health” interval = “5s” } } } }
Client Nomad Server Nomad Consul Consul Server Register Service Schedule App Monitor Health App 1 App N HASHICORP
Secret Distribution: API Keys DB Credentials SSL/TLS Certificates Nomad HASHICORP
example.nomad job “my-app" { … task “my-app" { env { DB_USERPASS = “foo:bar” } } }
Secure secret storage Dynamic secrets Leasing, renewal, and revocation Auditing Rich ACLs Vault Multiple client authentication methods
Login Vault Token Vault Token + Operation Op Response HASHICORP
example.nomad job “my-app" { … task “my-app" { env { VAULT_TOKEN = “b6a10b96-9060-11e6-9c6f-67a52bc6b8d3” } } }
example.nomad job “my-app" { … task “my-app" { vault { policies = [“my-app-role”] } } }
Submit Job + Vault Token Verify Vault Token Nomad Server Schedule App Client Generate + Renew Vault Token Nomad App 1 App N HASHICORP
Native Vault Integration No Secrets in Jobs No Secrets on Client Disk Minimize Trust Nomad HASHICORP
Stateless Stateful HASHICORP Stateful Applications
Stateless Stateful API Web Cache HASHICORP Stateful Applications
Stateless Stateful API HDFS Web Cassandra Cache MongoDB HASHICORP Stateful Applications
Stateless Stateful API HDFS *SQL Web Cassandra Cache MongoDB HASHICORP Stateful Applications
EASY MEDIUM HARD Stateless Stateful API HDFS *SQL Web Cassandra Cache MongoDB HASHICORP Stateful Applications
example.nomad job “my-app" { … task “my-app" { ephemeral_disk { sticky = true } } }
Moves data between tasks on the same machine HASHICORP
Copies data between tasks on di ff erent machines HASHICORP
Easily Deploy Apps: Declarative Jobs Flexible Workloads Consul Integration Vault Integration Sticky Volumes Nomad HASHICORP
Operationally Simple HASHICORP
Client Server HASHICORP
Built on Experience GOSSIP CONSENSUS
Cluster Management Gossip Based (P2P) Membership Failure Detection Serf Event System
Large Scale Production Hardened Simple Clustering and Federation Serf
Service Discovery Configuration Coordination (Locking) Central Servers + Distributed Clients Consul
Multi-Datacenter Raft Consensus Large Scale Production Hardened Consul
Operational Simplicity: Single Binary No Dependencies Highly Available Nomad HASHICORP
Built for Scale HASHICORP
Built on Experience Mature Libraries Proven Design Patterns GOSSIP CONSENSUS Lacking Scheduling Logic
Built on Research GOSSIP CONSENSUS
CLIENT CLIENT CLIENT DC1 DC2 DC3 RPC RPC RPC SERVER SERVER SERVER REPLICATION REPLICATION FORWARDING FORWARDING FOLLOWER LEADER FOLLOWER HASHICORP Single Region Architecture
REGION A SERVER SERVER SERVER REPLICATION REPLICATION FORWARDING FOLLOWER LEADER FOLLOWER GOSSIP REGION FORWARDING REGION B SERVER SERVER SERVER REPLICATION REPLICATION FORWARDING FOLLOWER LEADER FOLLOWER HASHICORP Multi Region Architecture
Region is Isolation Domain 1-N Datacenters Per Region Flexibility to do 1:1 (Consul) Scheduling Boundary Nomad HASHICORP
Hundreds of regions Tens of thousands of clients per region Thousands of jobs per region HASHICORP
Inspired by Google Omega Optimistic Concurrency State Coordination Service & Batch workloads Nomad Pluggable Architecture
Data Model NODE EVALUATION ALLOCATION JOB
Evaluation ~= State Change
Evaluations Create / Update / Delete Job Node Up / Node Down Allocation Failed / Finished
Evaluations SCHEDULER func(Evaluation) => []AllocationUpdates
Evaluations SCHEDULER func(Evaluation) => []AllocationUpdates Service, Batch, System
External Event EvaluaBon CreaBon EvaluaBon Queuing EvaluaBon Processing OpBmisBc CoordinaBon State Updates HASHICORP
Omega Architecture Optimistically Schedule 100’s of Jobs in Parallel Controls for Correctness Nomad HASHICORP
Nomad Million Container Challenge 1,000 Jobs 1,000 Tasks per Job 5,000 Hosts on GCE 1,000,000 Containers
“ 640 KB ought to be enough for anybody. – Bill Gates
2nd Largest Hedge Fund 18K Cores 5 Hours 2,200 Containers/second
Cluster Scheduler Easily Deploy Applications Operationally Simple Built for Scale Nomad HASHICORP
Thanks! Q/A HASHICORP
HASHICORP
Recommend
More recommend
