Leveraging Ambari to Build Comprehensive Management UIs For Your - PowerPoint PPT Presentation

Leveraging Ambari to Build Comprehensive Management UIs For Your Hadoop Applications by Christian Tzolov @christzolov

Whoami Christian Tzolov Technical Architect at Pivotal, BigData, Hadoop, SpringXD, Apache Committer, Crunch PMC member ctzolov@pivotal.io blog.tzolov.net @christzolov

How To Provision , Manage And Monitor Enterprise Applications At Scale ?

Contents • Ambari Overview • Stacks and Services (demo) • Blueprints (demo) • Views (demo) • Metrics & Monitoring • On the Cloud

Apache Ambari A platform to provision , manage and monitor Hadoop and Enterprise Applications

Deployment Topology admin node Ambari Web (REST API) Ambari Server cluster node 1 cluster node 2 cluster node N Ambari Agent Ambari Agent Ambari Agent

Special Powers Services & Stacks Monitoring Blueprints Views

Services & Stacks

Service Concepts • Stack - Set of (managed) Services and software packages for them • Service - Set of (distributed) Components • Component - Service building block of type Master, Slave, Client

Service Architecture

Service Layout • metainfo.xml - defines service, |_ services components and the |_ <service_name> metainfo.xml command management metrics.json scripts |_ configuration {configuration files} |_ package • configuration - service config |_ scripts files used in and deployed {command scripts} through the command scripts • package/scripts - commandScripts and templates.

Metainfo.xml <metainfo> <components> <schemaVersion/> <component> <services> <name>CUST_MASTER</name> <service> <displayName>Master</displayName> <category> MASTER </category> <name/> <cardinality>1</cardinality> <displayName/> < commandScript > <comment/> <script> scripts/master.py </script> <version/> <scriptType>PYTHON</scriptType> < components /> <timeout>600</timeout> <osSpecifics/> </ commandScript > <commandScript/> </component> <requiredServices/> ……… <configuration-dependencies/> </components> </service> </services> </metainfo>

Service / Command Script import sys import sys from resource_management import * from resource_management import * class Master (Script): class Slave (Script): def install (self, env): def install (self, env): def stop (self, env): def stop (self, env): def start (self, env): def start (self, env): def status (self, env): def status (self, env): def configure (self, env): def configure (self, env): if __name__ == "__main__": if __name__ == "__main__": Master().execute() Slave().execute()

Service / Demo

Ambari Blueprint

Why? • Installation Automation • Declarative definition of a cluster • Specify a Stack , the Component layout and the Configurations to materialize a cluster • REST API

Blueprint Flow REST API Blueprint Host Mapping (json) (json) Ambari Server Cluster 1 Cluster 2 Cluster Z node 1 node 2 node N . . . . . . . .

Blueprint Deploy POST /api/v1/ blueprints /my-blueprint POST /api/v1/ clusters /my-cluster { "Blueprints" : { { "stack_name" : "PHD", "blueprint" : "hdfs-hawq-blueprint", "stack_version" : "3.0" "default_password" : "secret-password", }, "host_groups" : [ "configurations" : [ { { "hawq-site" : { "name" : " management_masters ", "properties" : { "hawq.master.port" : "5532" } "hosts": [ { "fqdn":"ambari.localdomain" } ] } }, } { ], "name" : "masters_workers", "host_groups" : [ "hosts": [ { "fqdn":"phd1.localdomain" }, { "name" : " management_masters ", { "fqdn":"phd2.localdomain" }, "components" : [ { "fqdn":"phd3.localdomain" } ] { "name" : "NAMENODE" }, } { "name" : "HAWQMASTER" } ] ], } "cardinality" : "1" }, …… ]}

Blueprint REST API Cluster Logical Structure and /api/v1/blueprint configuration Physical cluster mapping. /api/v1/cluster Materializes a blueprint with real infrastructure /api/v1/clusters/<cluster name>? Exports blueprint for an existing format = blueprint cluster

Blueprint Demo

Ambari Views

View Components • Client Side - any client-side Ambari Web technologies Client Side VIEW PACKAGE (JAR) • Server Side - Java application, (js,html..) Servlet, a REST service or REST API ResourceProvider. context • View Package - JAR bundles of Server Side the view definition and resources (java) Ambari Server • View Context - Instance attributes and configuration

View Package view.xml <server packages + classes> index.html <supporting UI assets> WEB-INF (lib/*.jar)

View Context • On the Server Side ViewContext getUsername() : String • View-instance Config getViewName() : String getInstanceName() : String getProperties() : Map<String,String> putInstanceData(…) • View-instance Attributes getInstanceData(…) : String • Execution context, authenticated user principal

View Dashboard

View REST API GET /api/v1/views List deployed views List all View GET /api/v1/views/<View Name> instances GET /api/v1/views/< View Name >/versions/< View Show specific View Version > instance POST /api/v1/views/< View Name >/versions/< View Create new View Version >/instances/< Instance Name > Instance [{“ViewInstanceInfo":{ "properties" : { "dataworker.defaultFs":“webhdfs://namenode.host:50070" }}}]

Google Polymer https://www.polymer-project.org/1.0/

View Demo https://github.com/tzolov/ambari-webpage- embedder-view https://youtu.be/qaR_6OBeKsQ

Ambari Monitoring

Ambari Metrics System Collect, Aggregate and Serve System Metrics • Metric Collector - Collects and Aggregates • Metric Monitor - System-level metrics • Metric Hadoop Sinks - Service-level metrics

AMS Architecture

Ambari & Cloud

Virtualization & Cloud • Vagrant - PivotalHD (https://github.com/tzolov/ vagrant-pivotalhd) • Cloudbreak (http://sequenceiq.com/ cloudbreak/)

Thank You!