Big Data on Google Cloud Using Cloud Dataflow, BigQuery, and friends to process data “the Cloud way” William Vambenepe, Lead Product Manager for Big Data, Google Cloud Platform @vambenepe / vbp@google.com
Agenda 1 Big Data at Google 2 Managing data through its lifecycle 3 Google Cloud Dataflow 4 Friends of Cloud Dataflow: BigQuery, Pub/Sub, Hadoop/Spark... 5 Optimizing your time 6 References and follow-up
Big Data at Google
Building on Google’s infrastructure 1.5 million devices activated every day (over a billion devices) 6 billion hours watched every month (10h uploaded every minute) 20 billion pages crawled every day
Hardware and data center innovation
Software innovation BigQuery Cloud Dataflow MapReduce Flume Dremel MillWheel Spanner GFS Big Table Pregel Colossus 2002 2004 2006 2008 2010 2012 2013
Managing data through its lifecycle
Data lifecycle Google App Engine Real time analytics & alerts Cloud BigQuery Pub/Sub Analytics BigQuery (SQL) Stream Cloud Storage Logs (tables) Batch Google Cloud Analytics Dataflow Premium Cloud Google Cloud Dataflow Cloud Storage Storage (files)
Data usage organization maturity lifecycle Prescriptive + Predictive Exploratory Descriptive Predictive + Exploratory Descriptive Exploratory + Descriptive Descriptive
Supporting organizations with operational ease of use ● no administration ● most powerful tools in the easiest way ● constant experimentation with low risks & cost ● easy collaboration across teams and organizations ● low costs without requiring usage commitments ● best performance & virtually unlimited scale ● always on
Google Cloud Dataflow
Data lifecycle Google App Engine Real time analytics & alerts Cloud BigQuery Pub/Sub Analytics BigQuery (SQL) Stream Cloud Storage Logs (tables) Batch Google Cloud Analytics Dataflow Premium Cloud Google Cloud Dataflow Cloud Storage Storage (files)
What is Cloud Dataflow? Cloud Dataflow is Cloud Dataflow is a a collection of managed service SDKs for building for executing parallelized data parallelized data processing processing pipelines pipelines ↳ Download from GitHub: ↳ Use on Google Cloud: https://github.com/GoogleCloudPlatform/DataflowJavaSDK https://cloud.google.com/dataflow/
Cloud Dataflow SDK - Logical Model Pipeline{ Who => Inputs Unified programming What => Transforms model for both batch & Where => Windows stream processing. When => Watermarks + Triggers To => Outputs }
Cloud Dataflow Pipeline • A Direct Acyclic Graph of data processing transformations • Can be submitted to the Dataflow Service for optimization and execution or executed on an alternate runner e.g. Spark • May include multiple inputs and multiple outputs • May encompass many logical MapReduce operations • PCollections flow through the pipeline
Life of a Dataflow Pipeline Managed Service Work Manager Job Manager Graph optimization User Code & SDK Progress & Schedule Deploy & Logs Monitoring UI Google Cloud Platform
Worker Lifecycle Management throughout batch execution Schedule & Monitor Deploy Tear Down
Worker Optimization 100 mins. 65 mins. vs.
Continuous worker scaling for long-lived streaming pipelines 800 RPS 1,200 RPS 5,000 RPS 50 RPS time
Portability: Cloud Dataflow Runners • Run the same code in multiple modes using different runners • Direct Runner • For local, in-memory execution. • Great for developing and unit tests • Cloud Dataflow Service Runner • Runs on the fully-manage Dataflow Service • Your code runs distributed across GCE instances • Community sourced • Spark runner @ github.com/cloudera/spark-dataflow • Flink runner coming soon from dataArtisans The most productive and portable Data pipeline SDK.
Friends of Cloud Dataflow: BigQuery, Pub/Sub, Hadoop/Spark...
Data lifecycle Google App Engine Real time analytics & alerts Cloud BigQuery Pub/Sub Analytics BigQuery (SQL) Stream Cloud Storage Logs (tables) Batch Google Cloud Analytics Dataflow Premium Cloud Google Cloud Dataflow Cloud Storage Storage (files)
Cloud Pub/Sub Many-to-many asynchronous messaging Fast and reliable
BigQuery Ingest data via streaming (100K rows/second/table) or file loader ● Process interactive SQL queries on TB or PB of data ● Zero administration; just upload data and send queries ● Pay for storage and query separately, based on actual usage ● Non-technical analysts can drive queries on massive datasets using BI ● tools (e.g. Tableau) Highly Available: Data replication in multiple geographies. ● Secure and easy collaboration: access to data is controlled using ● customer-owned ACLs
Hadoop and Spark Name Local Master SSD Node Node (optional) GCS Connectors Connector PD SSD BigQuery Connector Work Nodes Work HDFS HDFS Work Nodes (optional) Nodes (optional) PD standard bdutil orchestration
Optimizing your time
Optimizing Your Time
Optimizing Your Time
Optimizing Your Time
Optimizing Your Time
Optimizing Your Time
Optimizing Your Time
Optimizing Your Time
Optimizing Your Time
Optimizing Your Time
References and follow-up
Getting Started Cloud Dataflow Service: https://cloud.google.com/dataflow ● Questions: https://stackoverflow.com/questions/tagged/google-cloud-dataflow ● SDK: https://github.com/GoogleCloudPlatform/DataflowJavaSDK ● BigQuery https://cloud.google.com/bigquery/ ● Contact me Cloud Pub/Sub Twitter: @vambenepe ● https://cloud.google.com/pubsub/ ● email: vbp@google.com ● Hadoop and Spark https://cloud.google.com/hadoop/ ●
Thank You! cloud.google.com
Recommend
More recommend
