Duy Hai DOAN @doanduyhai Who Am I ? Duy Hai DOAN Cassandra - - PowerPoint PPT Presentation

@doanduyhai

Apache Zeppelin, the missing GUI for your BigData eco-system

Duy Hai DOAN

@doanduyhai

Who Am I ?

Duy Hai DOAN Cassandra technical advocate

• talks, meetups, confs
• pen-source devs (Achilles, …)
• OSS Cassandra point of contact

☞ duy_hai.doan@datastax.com ☞ @doanduyhai

2

@doanduyhai

Datastax

• Founded in April 2010
• We contribute a lot to Apache Cassandra™
• 400+ customers (25 of the Fortune 100), 400+ employees
• Headquarter in San Francisco Bay area
• EU headquarter in London, offices in France and Germany
• Datastax Enterprise = OSS Cassandra + extra features

3

What is Apache Zeppelin ?

Presentation Architecture

@doanduyhai

Zeppelin Presentation

5

@doanduyhai

Demo

https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation

@doanduyhai

Zeppelin Architecture

Zeppelin Server

Zeppelin Engine

7

R E S T

WebSocket Spark Interpreter Group Spark SparkSQL Zeppelin Interpreter Factory Tajo Interpreter Flink Interpreter Cassandra Interpreter

JVM

JVM JVM JVM JVM

@doanduyhai

What does Zeppelin provide ?

Front-end & display system for free Generic back-end with REST APIs & WebSocket Pluggable interpreters system Task scheduler (à la CRON)

8

Zeppelin UI Layout

Notebook Paragraph UI elements

@doanduyhai

Demo

https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation

Zeppelin Display System

Raw, Table, HTML Available graphs View modes Dynamic form Iframe export

@doanduyhai

Demo

https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation

Interpreter system

Core interpreters Third-parties interpreters Interpreters conf & usage

@doanduyhai

Interpreter processing lifecycle

①

Receive input commands/data

• as raw text
• from form data

②

Process the input commands/data by the external back-end

③

Format the response using Zeppelin display system

④

Send response back to the Zeppelin engine

14

@doanduyhai

Core interpreters

• Spark (Spark core, SparkSQL/DataFrame, PySpark)
• Spark core = default (or %spark)
• SparkSQL = %sql
• Shell (%sh)
• Markdown (%md)
• AngularJS (%angular)

15

@doanduyhai

Third-parties interpreters

• Hive
• Phoenix
• Tajo
• Flink
• Ignite
• Lens
• Cassandra
• Geode
• PostgreSQL
• Kylin

16

@doanduyhai

Interpreter conf & usage

https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation

Writing An Interpreter

• How To

Simple interpreter example (AsciiDoc) Complex interpreter example (Cassandra)

@doanduyhai

Steps to write your own interpreter

• Create a class that extends Interpreter base class
• Register it in a static block
• Optionnally define default config params

19

static {

Interpreter.register("MyInterpreterName", MyClassName.class.getName()); }

static {

Interpreter.register("MyInterpreterName", MyClassName.class.getName(), new InterpreterPropertyBuilder() .add("property1", "default value", "Description of property1").build()); }

@doanduyhai

To register your interpreter as default

• Edit the enum ZeppelinConfiguration.ConfVars
• Add your interpreter FQCN in the property ZEPPELIN_INTERPRETERS

20

@doanduyhai

To register your interpreter in config files

• Create conf/zeppelin-site.xml from conf/zeppelin-site.xml.template
• Add your interpreter FQCN in the property zeppelin.interpreters

21

<property> <name>zeppelin.interpreters</name> <value>org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,

• rg.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,
• rg.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.shell.ShellInterpreter,
• rg.apache.zeppelin.hive.HiveInterpreter,com.me.MyNewInterpreter

</value> </property>

@doanduyhai

Simple AsciiDoc Interpreter

22

Zeppelin Server

AsciiDoc Interpreter JVM Zeppelin Engine

Raw Text Block Raw Text Block Converted To HTML HTML Output

① ② ③ ④ JVM

@doanduyhai

Simple interpreter (AsciiDoc)

https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation

@doanduyhai

Cassandra Interpreter Architecture

24

Cassandra Interpreter JVM

Display Results as HTML

① ② ⑤ Zeppelin Server

JVM

Raw Text Block Raw Text Block Cassandra

Cassandra Java Driver

③

Async CQL statements

④

Render HTML

⑥

@doanduyhai

Cassandra Interpreter Commands

25

Native CQL statements SELECT * FROM …; INSERT INTO …; … Schema commands DESCRIBE TABLE …; DESCRIBE KEYSPACE …; … Prepared statements Commands @prepare …; @bind …; @remove_prepared …; Help command HELP; Options Commands @consistency …; @retryPolicy …; @fetchSize …;

@doanduyhai

Complex interpreter (Cassandra)

https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation

Zeppelin future

• Roadmap

@doanduyhai

Roadmap & future

• More graph options (Map viz ZEPPELIN-157)
• Helium project, packaging Zeppelin view, logic (code) & resource into

Applications

• Interpreters packaging re-design
• ship & compile core interpreters only
• third-parties interpreters can be pulled from repository
• which interpreter is core ? Who will maintain ? Community….
• Integrate security (Apache Shiro, ZEPPELIN-53 )

28

@doanduyhai

Roadmap & future

• Out of incubation state to become 1st class Apache project

29

@doanduyhai

Q & R

@doanduyhai

Thank You

@doanduyhai duy_hai.doan@datastax.com http://zeppelin.incubator.apache.org/

@doanduyhai