Kerberos and Health Checks and Bare Metal, Oh My! Updates to - - PowerPoint PPT Presentation

Kerberos and Health Checks and Bare Metal, Oh My!

Updates to OpenStack Sahara in Newton

Updates to OpenStack Sahara in Newton

Vitaly Gridnev, Sahara PTL (Mirantis) Elise Gafford, Sahara Core (Red Hat) Nikita Konovalov, Sahara Core (Mirantis)

Agenda

1. Sahara overview 2. Health checks and management improvements 3. Kerberos integration for clusters 4. Image generation improvements 5. Bare metal clusters 6. What is NEW in NEWton 7. Q&A

Agenda

1. Sahara overview 2. Health checks and management improvements 3. Kerberos integration for clusters 4. Image generation improvements 5. Bare metal clusters 6. What is NEW in NEWton 7. Q&A

Sahara: The Use Cases

• Data Processing Cluster Management

○ On-demand, scalable, configurable, persistent clusters ○ Supports multiple plugins (Apache, Ambari, CDH, MapR...) ○ Integrates with Heat, Glance, Nova, Neutron, and Cinder

• EDP (Elastic Data Processing)

○ Supports multiple job types (Java, MR, Hive, Pig, Spark, Storm...) ○ Supports transient clusters (spin up, process, shut down) or persistent clusters ○ Integrates with Swift and/or Manila (optionally)

Sahara: The API

Sahara: The Project

• Cluster provisioning plugins:

○ Cloudera Distribution of Hadoop (using Cloudera Manager) ○ Hortonworks Data Platform (using Apache Ambari) ○ MapR ○ “Vanilla” Apache Hadoop, Spark, and Storm

• EDP job types:

○ MapReduce, Java, Hive, and Pig jobs (using Apache Oozie) ○ Spark, Spark Streaming, and Storm jobs (using Apache Spark and Apache Storm)

• Image packing repository (sahara-image-elements)
• Framework to validate Sahara installation (sahara-tests)
• UI plugin
• OpenStackClient plugin

Agenda

1. Sahara overview 2. Health checks and management improvements 3. Kerberos integration for clusters 4. Image generation improvements 5. Bare metal clusters 6. What is NEW in NEWton 7. Q&A

Event log for clusters

• Cluster events about provisioning: allows to

understand what is the current status of cluster provisioning, or reasons of failure

• Available since Newton for clusters created by

using Ambari

• Supported in CLI since Newton, with full dump of

all steps and events

Event log for clusters

Event log for clusters

Health checks for clusters

• Users are interested in monitoring cluster state

after cluster provisioning: vital for long living clusters

• Sahara in Liberty doesn't have any monitoring of

the health of cluster processes. A cluster can be broken or unavailable but Sahara will still think that it is in ACTIVE status.

Health checks for clusters

• Clusters health checks have been implemented since

Mitaka

• Available for clusters deployed using Ambari and

Cloudera Manager. Less availability for vanilla clusters

• Since Newton checks are available for the MapR

plugin

• Health results can be set to notify Ceilometer
• Easy to recheck health

Health checks for clusters

Health checks for clusters

Health checks for clusters

Health checks for clusters

Next steps are:

• More detailed health checks

○ Particular datanode/slave failure ○ No enough space in HDFS

• Suggestions/actions to repair health:

○ Datanode replacement ○ New nodes ○ Restarting services

• More flexible configuration of health checks (advanced health

checks, on disabling/enabling health checks for some reason)

Agenda

1. Sahara overview 2. Health checks and management improvements 3. Kerberos integration for clusters 4. Image generation improvements 5. Bare metal clusters 6. What is NEW in NEWton 7. Q&A

Security improvements

• Security is an important part of created clusters
• Previously security could be enabled only by

managers calling only Ambari and Cloudera Manager directly, but that leads to a situation in which Sahara will not perform auth operations, and EDP does not work

• Security is important not just for clusters, but for

Sahara itself

Security improvements

In Newton the following Kerberos security features were implemented:

• MIT KDC can be preconfigured (or an existing KDC can be used)
• Oozie client was re-implemented to support auth operations with Kerberos
• Spark job executions are also supported
• Keys are distributed on nodes for system users (hdfs, hadoop, spark)
• Supported for clusters deployed using Ambari and Cloudera Manager
• Note: Be sure that latest hadoop-swift jars are in place for Swift data sources!

Security improvements

Security improvements

• Bandit tests per commit
• Improved secret storage

(using Barbican and Castellan) was implemented in the previous release

Agenda

1. Sahara overview 2. Health checks and management improvements 3. Kerberos integration for clusters 4. Image generation improvements 5. Bare metal clusters 6. What is NEW in NEWton 7. Q&A

Where we were

Sahara had 2 flows that were relevant to image manipulation:

• Pre-Nova spawn image packing

○ Used sahara-image-elements repository to generate images (to store in Glance)

• Post-Nova spawn cluster generation from “clean” (OS-only) images

○ Logic maintained in Sahara process within plugins

• Pre-Configuration validation of images by plugins

○ Remember how I said we had 2 flows relevant to image manipulation? ○ We didn’t do this at all.

Where We Were: Problems

• Duplication of logic

○ Steps required for packing images and “clean” image clusters were often identical, but had to be expressed separately (in DIB and in Python).

• Poor validation

○ Plugins did not validate that images provided to them met their needs. ○ Failures due to image contents were late and sometimes difficult to understand.

• Poor encapsulation

○ Image generation and cluster provisioning logic for any one plugin are really one application ○ Maintaining them in two places allows versionitis and dependency problems ○ Having one monolithic repo for all plugins makes them less pluggable

Our Dream Implementation

• All flows share common logic:

○ Image packing ○ Image validation ○ Clean image cluster gen

• Image manipulation is stored and versioned within plugins
• The user can still generate images with a CLI...
• But they can also use an API to generate images in clean build environments
• ... And both dev test cycles and user retries are as quick and painless as

possible

The plan

1. Build a validation engine that ensures that images meet a specification

a. YAML-based spec definition

2. Extend that engine to optionally modify images to spec 3. Build a CLI to expose this functionality 4. Create and test specifications for each plugin to support this method 5. Deprecate sahara-image-elements (only when this method proves stable) 6. Build an API to:

a. Spawn a clean tenant-plane image build environment b. Download a base image from Glance and modify it to spec c. Push the new image back to Glance and register it for use by Sahara

Where we are

1. Build a validation engine that ensures that images meet a specification

a. YAML-based spec definition

2. Extend that engine to optionally modify images to spec 3. Build a CLI to expose this functionality 4. Create and test specifications for each plugin to support this method 5. Deprecate sahara-image-elements (only when this method proves stable) 6. Build an API to:

a. Spawn a clean tenant-plane image build environment b. Download a base image from Glance and modify it to spec c. Push the new image back to Glance and register it for use by Sahara

What it looks like: the specs

• YAML-based definitions
• Argument definitions for

configurability

• Idempotent resource

declarations ○ Scripts must be written idempotently, as always in resource declarations

• Logical control operators (any,

all, os_case, etc.)

What it looks like: the CLI

Command structure: sahara-image-pack --image ./image.qcow2 PLUGIN VERSION [plugin arguments] Features:

• Auto-generates help text from arguments
• Idempotent and modifies images in-place

○ Very fast test cycles and retries

• Allows freeform bash scripts and more

structured resources

○ Though it’s on you to make your scripts idempotent

• Test-only mode to validate without change

What it’s doing

The images module runs a sequence

• f steps against a remote machine
• Validation uses the Sahara SSH remote in

read-only mode

• Clean image gen uses the SSH remote
• Image packing uses a libguestfs Python

API image handle

All three use the same logic, contained in the appropriate plugin Plugin implementation targeting O!

Agenda

1. Sahara overview 2. Health checks and management improvements 3. Kerberos integration for clusters 4. Image generation improvements 5. Bare metal clusters 6. What is NEW in NEWton 7. Q&A

Ironic integration

Why should you run Bare Metal in OpenStack:

• Big Data workload originates from Bare Metal installations
• Quick cluster scalability may have lower priority than a long running stability

and persistence

• Best performance by design, no virtualization overhead
• The ability to manage a baremetal cluster with the OpenStack API

Bare Metal compared to Virtualized

Bare metal (Ironic) Virtual Machines Cluster size flexibility

Dedicating nodes completely. Flavor based scheduling

Resource utilization

The host is 100% utilized. KVM has memory overhead. Other VM may abuse host’s resources.

Data locality

Data is accessible directly from the local disks. Locality may be achieved by proper resource scheduling

Live migration

A host may be lost completely. Supported for some target daemons

Some tips before running Bare Metal

• Scheduling is not trivial. The Cloud operator may need to specify additional

Flavors, Availability Zones, or other metadata

• Storage is not backed by Cinder for Bare Metal

○ Sahara does disk discover on it’s own ○ Disks are different from the on w/o root mount are going to be dedicated to HDFS

• Non-standard hardware will require drivers built into the provisioning image
• Network tenant isolation is achievable through manual hardware switch

configurations

Agenda

1. Sahara overview 2. Health checks and management improvements 3. Kerberos integration for clusters 4. Image generation improvements 5. Bare metal clusters 6. What is NEW in NEWton 7. Q&A

What is NEW in NEWton

• Designate integration;
• API Improvements: pagination for list operations, API to

manage/enable/disable plugins;

• New plugin versions

○ HDP 2.4 supported ○ MapR 5.2.0 ○ CDH 5.7.x ○ Vanilla + Spark on YARN

What is NEW in Newton

• Sahara tests framework to validate

environment readiness for Sahara’s clusters

○ Sahara tempest plugin with more tests (CLI, API) ○ Sahara scenario framework with a bunch of templates ○ Published on PyPi https://pypi.python.org/pypi/sahara-tests

Q&A

Useful links and materials

• Sahara wiki https://wiki.openstack.org/wiki/Sahara
• Sahara specs https://specs.openstack.org/openstack/sahara-specs/
• Sahara docs http://docs.openstack.org/developer/sahara/
• Sahara images http://sahara-files.mirantis.com/images/upstream/newton/