Research in Production Clouds Designed for Transition intelligent - - PowerPoint PPT Presentation

Research in Production

Mike May, Technology Director

Clouds Designed for Transition

intelligent architectures and big data science

WHO I AM

• Mike May, Technical Director

– 30+ cloud deployments

• 4 production US Government clouds
• 6 simultaneous research DARPA clouds

– Background

• Cybersecurity
• HPC Systems Engineering
• Stacker since 2013

2

WHAT WE DO

• Supporting 10 Active R&D Programs

– Mostly DARPA programs – Design and deploy upstream IaaS

• OpenStack
• Mesos
• Kubernetes
• Burst support to public clouds

– Drastically diverse research goals

• Data science and data analytic heavy

3

WHAT WE DEPLOY

4

BANANA(🍍) FOR SCALE

• 1 DARPA Program Cluster (OpenStack and Mesos)

– 476 Raw CPU cores (952 Threads/vCPUs) – 13TB RAM; never overprovisioned – 2PB Raw Disk – 14 GPU Nodes – 48 Pascal GPUs

• 172032 CUDA Cores
• 576GB of VRAM

– Bare metal Mesos with burst to OpenStack – GPU development VMs available in OpenStack – GPU batch job support in Mesos – 100% Open Source Tools

5

HOW IS IT USED

• Seamless development to production experience

– Hardware is shifted between IaaS offerings as needed – Development heavy start, production heavy transition

• Simultaneous provisioned and batch job support
• CI/CD processes automatically promote work to

relevant cluster resources

– Provided to users automatically and they have full control – Fire and forget methodology; fail fast

• L2 isolation by default

6

SYSTEM LIMITS

• CPU overprovisioning is never >8x (which is a

lot)

– CPU NOPs are our enemy too – We collect metadata on performance; metadata is collected on boxes that are currently not

• verprovisioned

– Per-program level – Case-by-case

7

SYSTEM LIMITS (CONT.)

• RAM is NEVER overprovisioned

– Problems bubbled up as bare metal OS issues

• GPUs are (painfully) special

– In and out of batch processing pipelines – Obvious but important: development and experiments change use case and needs

8

STARTING WITH A BASELINE

• We needed a baseline that others could

reproduce locally

• Fuel was a great start because of the web

interface that made the process much easier to ingest

9

CUSTOMIZE OFF OF BASELINE

• Ansible supported all customizations applied

after a baseline deployment

• ”Program-public” for all to see

10

CLOUD OPS

• Cloud administration

– Configuration management

• “Ansiblize” all the things
• Idempotency is key

– Automation

• “(Almost) any task I have done more than once is to be

automated/scripted”

– Easier said than done

11

SYSTEMS FROM CODE

• All it takes to build an OpenStack base image

– GitLab – Packer – Ansible

12

A LITTLE CODE

13

CLOUDS AS CODE

CODE FOR OUR CURRENT DEPLOYMENTS

Every management and service task is captured and reviewable by the entire team.

14

SELF-SERVICE PROXY WITH AUTHENTICATION

15

AUTHENTICATION USER DRIVEN

LESSONS LEARNED

• Putting off automation reduces the chance you will ever

do it

• Monitoring is hard to do right but powerful to

understand user’s interactions with services

• Ground truth / root cause EVERYTHING

– Issues, alerts, crashes, user reports

• Researchers are biased (and so are admins and
• perators)
• Evacuation must always be an option

– Resource planning

• Document and train by default

16

17

THANK YOU!