SLIDE 1
Armada
Kubernetes multi-cluster batch scheduler 1 / 12
Introduction
G-Research - Compute Platform Engineering Team We build tools around HTCondor clusters and Kubernetes clusters 2 / 12
Armada Kubernetes multi-cluster batch scheduler 1 / 12 - - PowerPoint PPT Presentation
Armada Kubernetes multi-cluster batch scheduler 1 / 12 - - PowerPoint PPT Presentation
Armada Kubernetes multi-cluster batch scheduler 1 / 12 Introduction G-Research - Compute Platform Engineering Team We build tools around HTCondor clusters and Kubernetes clusters 2 / 12 Agenda 1. Project requirements 2. Current options 3.
SLIDE 2
SLIDE 3
Agenda
- 1. Project requirements
- 2. Current options
- 3. What is Armada
- 4. Armada architecture
- 5. Armada from user perspective
- 6. Demo
- 7. Questions
3 / 12
SLIDE 4
Project requirements
Experimental project to explore how to use Kubernetes for HPC Jobs. Easily handle large queues of jobs (million +) Enforce fair share over time Failure of few clusters should not cause an outage Reasonable latency between job submission and job start (in
- rder of 10s)
Maximize utilization of the cluster Smart queue instead of scheduler - implement only minimum logic needed on global level, let cluster scheduler do its own work. All components should be highly available 4 / 12
SLIDE 5
Current options
Existing schedulers: default scheduler, Kube-batch / Volcano, Poseidon (Firmanent) All focus on scheduling within one cluster Ocial limit for one cluster is 5000 nodes, but in practice this scale is dicult to achieve 5 / 12
SLIDE 6
What is Armada
Armada is job queuing system for multiple Kubernetes clusters. It maintains fair share over time similarly to HTCondor. It can handle large amount of queued jobs. Allow adding and removing clusters from the system. 6 / 12
SLIDE 7
Armada architecture
Submit Kubernetes Api Server Kubernetes Cluster Queue 1 Queue 2 Queue 3 Accounting Watch nodes & pods Create or delete pods Get workloads to schedule Armada Executor API Report events Events Recording Report Usage Events Database Jobs database Armada Server
Pull based model Clusters can be removed or added without disruptions 7 / 12
SLIDE 8
Armada from user perspective
Queue: Represent user or project, used to mantain fair share over time, has priority factor Job: Unit of work to be run (describe as Kubernetes PodSpec) Job Set: Group of related jobs, api allows observing progress of job set together
jobs:
- queue: test
priority: 1 jobSetId: my-project podSpec: ... kubernetes pod spec ... $ armadactl submit jobs.yaml $ armadactl watch my-project
8 / 12
SLIDE 9
Demo !
Running 5 clusters 99 worker nodes each (3 control plane nodes). Live: Queues Worker Clusters Snapshot: Queues Worker Clusters 9 / 12
SLIDE 10
Roadmap
Finish core functionality (cancellation, job priority change) Improve scheduling (better fairness, handling heterogenous hardware & jobs, preemption) Better authentication / authorization (JWT support) Improve user tooling (armadactl, potentially UI) Integration with other tooling (Argo, ...) How to handle k8s namespaces & secrets GPUs 10 / 12
SLIDE 11
Questions ?
11 / 12
SLIDE 12
Thank you.
Compute Platform Engineering at G-Research. Jan Kaspar, James Murkin, Jamie Poole. https://www.gresearch.co.uk 12 / 12