Moving up the stack with Stacktician Amogh Vasekar amoghvk@gmail.com - - PowerPoint PPT Presentation

▶
moving up the stack with stacktician
SMART_READER_LITE
LIVE PREVIEW

Moving up the stack with Stacktician Amogh Vasekar amoghvk@gmail.com - - PowerPoint PPT Presentation

Apr 10, 2024 174 likes •501 views

Moving up the stack with Stacktician Amogh Vasekar amoghvk@gmail.com Agenda Introduction History Architecture and Implementation Current State Enhancements Agenda Introduction History Architecture and Implementation

slide-1
SLIDE 1

Moving up the stack with Stacktician

Amogh Vasekar amoghvk@gmail.com

slide-2
SLIDE 2

Agenda

  • Introduction
  • History
  • Architecture and Implementation
  • Current State
  • Enhancements
slide-3
SLIDE 3

Agenda

  • Introduction
  • History
  • Architecture and Implementation
  • Current State
  • Enhancements
slide-4
SLIDE 4

Introduction

What is Amazon CloudFormation?

  • Create and manage collection of related AWS resources
  • Typically, the resources will form your whole application “stack”

In essence, “Infrastructure-as-code”

  • JSON file
  • Describe resources
  • Describe dependencies
  • Readable, reusable, reviewable
slide-5
SLIDE 5

Sample JSON template

slide-6
SLIDE 6

Usefulness

  • Single click application deployment for multiple users
  • Transparent, reusable
  • Ease of resource management and dependencies
  • Atomic deployments
  • Thus, migrate new business applications to “cloud”
slide-7
SLIDE 7

Simply put, CloudFormation for CloudStack J Web interface for template and stack management

slide-8
SLIDE 8

Agenda

  • Introduction
  • History
  • Architecture and Implementation
  • Current State
  • Enhancements
slide-9
SLIDE 9

History

  • Started by @chiradeep
  • Introduced at CCC-13 as a prototype
  • Execute AWS CloudFormation templates on CloudStack *AS IS*

AS IS*

  • Pr

Problem – low fidelity between A

  • blem – low fidelity between AWS and

WS and CloudStack CloudStack r resour esource ce types ( types (mor

more on this later e on this later)

slide-10
SLIDE 10

Agenda

  • Introduction
  • History
  • Architecture and Implementation
  • Current State
  • Enhancements
slide-11
SLIDE 11

Architecture

Two components

StackMate

  • Execution engine for running the templates and interfacing with CloudStack

Stacktician

  • Web interface
  • Embeds StackMate underneath
slide-12
SLIDE 12

StackMate in 2 mins

  • Treat template = stack definition as workflow
  • Execute workflow by making appropriate CloudStack API calls
  • Core engine for executing templates, does the heavy lifting
  • Command line tool
slide-13
SLIDE 13

Stacktician in 2 mins

  • Web UI for StackMate
  • Backed by database for persistence
  • Uses Ruby meta-programming to interface with StackMate
slide-14
SLIDE 14

Where we were last time…

  • Resolve complex resource dependencies
  • Intrinsic functions
  • Wait Conditions
  • Create simple resources like AWS::Instance and AWS::SecurityGroup

using StackMate

slide-15
SLIDE 15

AWS templates on CloudStack

  • Simple resources work well
  • Too many differences in more complex resources like VPC and networks

Ø No route tables

  • Semantic differences between APIs for same resources

Ø Different types of values for parameters

  • Dependency resolution for CloudStack using AWS templates became too complex

Ø Sometimes, two AWS APIs act as a single one in CloudStack Ø CloudStack networking renders AWS options like DhcpOptions unnecessary Ø Too many NoOp resources used as work-around for above

slide-16
SLIDE 16

CloudStack Namespace

  • Instead create CloudStack specific namespace!
  • No more mapping of resources from AWS namespace to CloudStack

space

  • Adds the ability to validate API parameters
  • Every CloudStack resource becomes a “participant” in the workflow

execution

  • Essentially need to call create and delete APIs
slide-17
SLIDE 17

CloudStack Namespace

  • Problems

Ø New resources added frequently (e.g tags, UCS, Nicira) Ø APIs evolve to support new parameters etc. Ø Too many CloudStack resources to maintain (~51)

  • Solution

Ø Use API discovery Ø Auto generate code Ø Use API information to validate all required parameters exist / format of parameters

slide-18
SLIDE 18

StackMate participants

  • Bootstrap StackMate
  • Publish CloudStack version specific gem
slide-19
SLIDE 19

Errors and rollback

  • Want stack creation to be atomic
  • But, provide an option to user to disable rollback (e.g. for debugging)

Ø Catch resource creation errors, and initiate rollback Ø Execute deletion in reverse order of dependencies Ø Skip any resources not yet created Ø If error while deleting, notify user (log for StackMate, UI status for Stacktician) Ø Stacktician injects additional code in StackMate for DB management

slide-20
SLIDE 20

Metadata server

  • Resources, like VMs, can launch additional initialization scripts upon configuration

Ø E.g. – Launch a script via “userdata” parameter

  • Typically, script needs additional metadata to work with
  • Metadata is specific to a resource

Request metadata

slide-21
SLIDE 21

Metadata server

And expose retrieval as API

slide-22
SLIDE 22

Metadata Server

  • StackMate

In memory metadata, embeds a Sinatra server for waitcondition and metadata

  • Stacktician

Rails serves metadata, persisted in DB

(Special thanks to Simon Waterhouse for integrating cfn-init scripts and creating scripts for Windows)

slide-23
SLIDE 23

Stacktician Scaling Improvements

  • Ruby MRI has global interpreter lock. Eventually, allowed for only one

stack execution thread to run at a time Ø Use JRuby container, run in JRuby compatible server

  • Async wait for stack : one thread per stack launch would overwhelm

server Ø GetBack gem for managing thread pool

slide-24
SLIDE 24

Delete Stack

Stacktician

  • Resource IDs in database
  • Execute delete APIs in reverse order of dependency creation
  • Not fully tested yet

StackMate

  • No DB, no IDs
  • Use resource tags to figure out resources, and delete
  • In progress
slide-25
SLIDE 25

Improvements

StackMate

  • Better logging, log level control
  • Light-weight CloudStack client for making API calls

Stacktician

  • Supports REST-based API for stack operations, querying and manipulation
  • Command line tools for all the above operations
  • sm-create-stack, sm-create-template etc
slide-26
SLIDE 26

Agenda

  • Introduction
  • History
  • Architecture and Implementation
  • Current State
  • Enhancements
slide-27
SLIDE 27

Demo

slide-28
SLIDE 28

Agenda

  • Introduction
  • History
  • Architecture and Implementation
  • Current State
  • Enhancements
slide-29
SLIDE 29

Future enhancements

Better plugin architecture

  • Allow creation of custom type of resources
  • Interface with other types of Clouds – GCE, AWS etc.
  • Allow for scenarios like resource A on CloudStack, B on AWS within single stack
  • Currently only in StackMate, and very rudimentary. Enforces some basic constraints,

uses tag “stackmate_participant = true” for discovery Ø Bad plugins rejected at run-time

  • Needs more work to make robust
slide-30
SLIDE 30

Future enhancements

  • Nested stacks

Ø Treat stack as first class object. Template can contain another stack definition as a resource Ø Works in StackMate, disabled for lack of good implementation

  • Support conditional functions in template (e.g Fn:And, Fn:Or, Fn:If)
  • Support update stack

Ø Semantics to be well defined

slide-31
SLIDE 31

Q & A

Questions? Thoughts? Comments? Suggestions? https://github.com/stackmate https://github.com/stackmate Feel free to log issues – more the better! Big thanks to Chiradeep for all the help