Leveraging Ambari to Build Comprehensive Management UIs For Your - - PowerPoint PPT Presentation

Leveraging Ambari to Build Comprehensive Management UIs For Your Hadoop Applications

by Christian Tzolov @christzolov

Whoami

Christian Tzolov Technical Architect at Pivotal, BigData, Hadoop, SpringXD, Apache Committer, Crunch PMC member ctzolov@pivotal.io blog.tzolov.net @christzolov

How To Provision, Manage And Monitor Enterprise Applications At Scale?

Contents

• Ambari Overview
• Stacks and Services (demo)
• Blueprints (demo)
• Views (demo)
• Metrics & Monitoring
• On the Cloud

Apache Ambari

A platform to provision, manage and monitor Hadoop and Enterprise Applications

Deployment Topology

Ambari Web (REST API) Ambari Server

admin node

Ambari Agent

cluster node 1

Ambari Agent

cluster node 2

Ambari Agent

cluster node N

Special Powers

Services & Stacks Blueprints Views Monitoring

Services & Stacks

Service Concepts

• Stack - Set of (managed) Services and software

packages for them

• Service - Set of (distributed) Components
• Component - Service building block of type

Master, Slave, Client

Service Architecture

Service Layout

|_ services |_ <service_name> metainfo.xml metrics.json |_ configuration {configuration files} |_ package |_ scripts {command scripts}

• metainfo.xml - defines service,

components and the command management scripts

• configuration - service config

files used in and deployed through the command scripts

• package/scripts -

commandScripts and templates.

Metainfo.xml

<metainfo> <schemaVersion/> <services> <service> <name/> <displayName/> <comment/> <version/> <components/> <osSpecifics/> <commandScript/> <requiredServices/> <configuration-dependencies/> </service> </services> </metainfo>

<components> <component> <name>CUST_MASTER</name> <displayName>Master</displayName> <category>MASTER</category> <cardinality>1</cardinality> <commandScript> <script>scripts/master.py</script> <scriptType>PYTHON</scriptType> <timeout>600</timeout> </commandScript> </component> ……… </components>

Service / Command Script

import sys from resource_management import * class Master(Script): def install(self, env): def stop(self, env): def start(self, env): def status(self, env): def configure(self, env): if __name__ == "__main__": Master().execute() import sys from resource_management import * class Slave(Script): def install(self, env): def stop(self, env): def start(self, env): def status(self, env): def configure(self, env): if __name__ == "__main__": Slave().execute()

Service / Demo

Ambari Blueprint

Why?

• Installation Automation
• Declarative definition of a cluster
• Specify a Stack, the Component layout and the

Configurations to materialize a cluster

• REST API

Blueprint Flow

Ambari Server node 1

Cluster 1

node 2 node N

Cluster 2

. . . .

Cluster Z

. . . . Blueprint (json) Host Mapping (json)

REST API

Blueprint Deploy

{ "Blueprints" : { "stack_name" : "PHD", "stack_version" : "3.0" }, "configurations" : [ { "hawq-site" : { "properties" : { "hawq.master.port" : "5532" } } } ], "host_groups" : [ { "name" : "management_masters", "components" : [ { "name" : "NAMENODE" }, { "name" : "HAWQMASTER" } ], "cardinality" : "1" }, …… ]} POST /api/v1/blueprints/my-blueprint { "blueprint" : "hdfs-hawq-blueprint", "default_password" : "secret-password", "host_groups" : [ { "name" : "management_masters", "hosts": [ { "fqdn":"ambari.localdomain" } ] }, { "name" : "masters_workers", "hosts": [ { "fqdn":"phd1.localdomain" }, { "fqdn":"phd2.localdomain" }, { "fqdn":"phd3.localdomain" } ] } ] } POST /api/v1/clusters/my-cluster

Blueprint REST API

/api/v1/blueprint Cluster Logical Structure and configuration /api/v1/cluster Physical cluster mapping. Materializes a blueprint with real infrastructure /api/v1/clusters/<cluster name>? format=blueprint Exports blueprint for an existing cluster

Blueprint Demo

Ambari Views

View Components

Ambari Server Ambari Web Server Side (java) Client Side (js,html..)

REST API VIEW PACKAGE (JAR)

• Client Side - any client-side

technologies

• Server Side - Java application,

Servlet, a REST service or ResourceProvider.

• View Package - JAR bundles of

the view definition and resources

• View Context - Instance

attributes and configuration

context

View Package

view.xml <server packages + classes> index.html <supporting UI assets> WEB-INF (lib/*.jar)

View Context

• On the Server Side
• View-instance Config
• View-instance Attributes
• Execution context, authenticated user principal

ViewContext getUsername() : String getViewName() : String getInstanceName() : String getProperties() : Map<String,String> putInstanceData(…) getInstanceData(…) : String

View Dashboard

View REST API

GET /api/v1/views List deployed views GET /api/v1/views/<View Name> List all View instances GET /api/v1/views/<View Name>/versions/<View Version> Show specific View instance POST /api/v1/views/<View Name>/versions/<View Version>/instances/<Instance Name>

[{“ViewInstanceInfo":{ "properties" : { "dataworker.defaultFs":“webhdfs://namenode.host:50070" }}}]

Create new View Instance

Google Polymer

https://www.polymer-project.org/1.0/

View Demo

https://github.com/tzolov/ambari-webpage- embedder-view https://youtu.be/qaR_6OBeKsQ

Ambari Monitoring

Ambari Metrics System

Collect, Aggregate and Serve System Metrics

• Metric Collector - Collects and Aggregates
• Metric Monitor - System-level metrics
• Metric Hadoop Sinks - Service-level metrics

AMS Architecture

Ambari & Cloud

Virtualization & Cloud

• Vagrant - PivotalHD

(https://github.com/tzolov/ vagrant-pivotalhd)

• Cloudbreak

(http://sequenceiq.com/ cloudbreak/)

Thank You!