Design Patterns Leveraging Spark in PDI Chris Skirde Pentaho - - PowerPoint PPT Presentation

Design Patterns Leveraging Spark in PDI

Chris Skirde Pentaho Director of Sales Engineering, Hitachi Vantara Rakesh Saha Pentaho Senior Product Manager, Hitachi Vantara

Quiz Time!

• What is Spark?
• A. A good way to start a fire.
• B. Necessary for a well running internal combustion engine.
• C. Fast and general purpose engine for large-scale data processing.
• D. All of the above.
• True or False, Pentaho supports Spark?
• Who is using Spark today (with or without Pentaho)?

Agenda

• Introduction to Spark
• Common design patterns
• How to leverage Spark with Pentaho

Introduction to Spark

• Why are we interested?
• What is it really?
• What’s been done?

Spark Application Architecture

Daemon PDI/Server

What Do Those Applications Have in Common?

Common Design Patterns

• Filter/Organize
• Join
• Sum
• Transform/Enrich
• Query
• Machine Learning/Data Science

Filter/ Organize

Join

Sum (and Other Aggregations)

Transform/Enrich

• Any step you like!

Query – Easy!

• Cloudera use Hive-on-Spark with Hive2
• Hortonworks use SparkSQL via Simba

Machine Learning/Data Science

Recap

What we covered today:

• Reviewed what Spark is and why organizations are adopting it
• Discussed several common data integration design patterns
• Linked those design patterns to Pentaho features for you to try

Questions?

Next Steps

Want to learn more?

• “Meet the Experts” Matt Casters and Mark Hall!
• Adaptive Execution Layer http://www.pentaho.com/blog/introducing-adaptive-

execution-layer-spark-architecture

• SQL on Spark http://www.pentaho.com/blog/operationalize-spark-big-data-

newest-enhancements