Big Telco, Bigger DW Demands: Moving Towards SQL-on-Hadoop Keuntae - PowerPoint PPT Presentation

Big Telco, Bigger DW Demands: Moving Towards SQL-on-Hadoop

Keuntae Park • IT Manager of SK Telecom, South Korea’s largest wireless communications provider • Work on commercial products (~’12) – T-FS: Distributed File System – Windows compatible layer on TimOS – T-MR: on-demand MapReduce service like E-MR • Open source activity (‘13~) – Committer of Apache Tajo project

Overview • Background – Telco requirements • Before Tajo – Commercial product – Open source (Hadoop) outsourcing • After Tajo – Issues & solutions – Performance • win-win between community and company • Future Works

Telco data characteristics • Huge amount of data – 40 TB/day (compressed) – 15 PB (estimated, end of 2014) • Report & OLAP ad-hoc query – Filtering – Summary – BI tools

Requirements - different size, different speed Filtering & Data re- Summary BI report Ad-hoc Query aggregation construction accumulated daily sum of entire Target mart data summary data for 5 minutes filtered data summary data every 5 daily or non-regularly Frequency ah-hoc ah-hoc minutes monthly (rare) Amount of hundreds of tens of tens of terabytes petabytes data terabytes gigabytes terabytes Response within a no strict within two within a hour within a hour time minute deadline seconds

Previous approach - DBMS based on MPP DBMS

Previous approach - DBMS Too Expensive Not Scalable based on MPP DBMS

Previous approach - DBMS Too Expensive Not Scalable based on MPP DBMS

Previous approach - DBMS Too Expensive Not Scalable based on MPP DBMS

Previous approach - Hadoop(MapReduce, Hive) + DBMS Hadoop MPP DBMS

Previous approach - Hadoop(MapReduce, Hive) + DBMS Working (but…) Hadoop MPP DBMS

Still has Problems • Hadoop outsourcing – quality of outcome is not good (actually bad) – communication overhead – hard to reflect requirements on open source • Data Warehouse and Mart becomes bigger

Solution - Tajo!! • It can replace both DBMS and Hadoop – High throughput for batch processing – Low latency for ad-hoc queries – ANSI SQL compatible • Can do by myself – very open community • easily make issues about what I really need – fast growing • issues solved very fast

About Tajo • Tajo (since 2010) – Big Data Warehouse System on Hadoop – Apache top-level project (entered the ASF in March 2013) • Features – SQL standard compliance – Fully distributed SQL query processing – HDFS as a primary storage – Relational model (will be extended to nested model in the future) – ETL as well as low-latency relational query processing (100 ms ~) • News – 0.2-incubating: released November 2013 – graduation to top-level: April 2014

Tajo logical optimizer • Cost-based join ordering • Projection/Filter push down & Duplicated expression removal aggr_sum1 aggr_sum2 GroupBy Filter sel_> sel_< Projection Join ID QTY Date ID Tax Price Table A Table B

Tajo logical optimizer • Cost-based join ordering • Projection/Filter push down & Duplicated expression removal aggr_sum1 aggr_sum1 aggr_sum2 GroupBy aggr_sum2 GroupBy Filter sel_> sel_< Join Projection sel_> sel_< Filter Join Projection ID QTY Date ID Tax Price ID QTY Date ID Price Tax Table A Table B Table A Table B

Tajo progressive optimization • dynamically adjust number of tasks input data • estimate data size 
 at planning time execution block • check size and adjust plan 
 intermediate data unknown priorly at execution time … shuffled shuffled shuffled how many tasks 
 data data data • shuffle intermediate data (and workers)? � … over workers uniformly � execution block

Tajo progressive optimization • dynamically adjust join order or type Hash-Join Hash-Join

Tajo progressive optimization • dynamically adjust join order or type Hash-Join Broadcast-Join Hash-Join

Tajo - what is improved past 9 months ? • Resource Manager • Scheduler & Storage Manager • Data types & Functions • SQL Interface • Management

Tajo resource manager • Fine resource allocation Tajo Master Tajo Worker 
 (as a query master) Tajo Worker 
 Tajo Worker 
 Tajo Worker 
 (as a worker) (as a worker) (as a worker) TAJO-127 without YARN

Tajo resource manager • Fine resource allocation Tajo Master Tajo Master Tajo Worker 
 Query Master (as a query master) Tajo Worker 
 Tajo Worker 
 Tajo Worker 
 Tajo Worker 
 Tajo Worker 
 (as a worker) (as a worker) (as a worker) Tajo Worker (as a worker) (as a worker) TAJO-127 TAJO-275 without YARN separating Query master

Tajo resource manager • Fine resource allocation Tajo Master Tajo Master Tajo Master Tajo Worker 
 Query Master Query Master (as a query master) Tajo Worker 
 Tajo Worker 
 Tajo Worker 
 Tajo Worker 
 Tajo Worker Tajo Worker Tajo Worker 
 (as a worker) (as a worker) Tajo Worker Tajo Worker (as a worker) Tajo Worker (as a worker) Tajo Worker Tajo Worker (I/O-intensive) (I/O-intensive) (as a worker) (I/O-intensive) (I/O-intensive) (I/O-intensive) (CPU/memory) TAJO-127 TAJO-275 TAJO-317 without YARN separating Query master elaborate resource allocation

Scheduler & Storage manager • disk-aware scheduling (volume info from HDFS-3672) Tajo Worker Tajo Worker Thread Tajo Worker Thread Thread Tajo Worker Tajo Worker Thread Tajo Worker Thread Thread

Scheduler & Storage manager • disk-aware scheduling (volume info from HDFS-3672) Tajo Worker Tajo Worker Tajo Worker Tajo Worker Tajo Worker Tajo Worker Tajo Worker Thread Thread Tajo Worker Thread Thread Thread Thread Tajo Worker Thread Thread Thread Tajo Worker Tajo Worker Thread Tajo Worker Thread Thread Storage Manager TAJO-84 considering disk load balance TAJO-178 asynchronous scan

Scheduler & Storage manager • disk-aware scheduling (volume info from HDFS-3672) Tajo Worker Tajo Worker Tajo Worker Tajo Worker Tajo Worker Tajo Worker Tajo Worker Thread Thread Tajo Worker Thread Thread Thread Thread Tajo Worker Thread Thread Thread Tajo Worker Tajo Worker Thread Tajo Worker Thread TAJO-134 Thread text compression Storage (gzip, snappy, lz4, bzip2) Manager TAJO-200 RCFile � TAJO-30 Parquet TAJO-84 TAJO-435 considering disk load balance intermediate file TAJO-178 asynchronous scan

Functions & data types • supporting more functions and UDFs function1 function2 Tajo Master function3 registered at startup (class name is coded in source)

Functions & data types • supporting more functions and UDFs function function function1 Tajo Master function2 Tajo Master user defined user defined function3 function function @Description( functionName = "to_timestamp", description = "Convert UNIX epoch to time stamp", registered at startup example = "> SELECT to_timestamp(1389071574);\n" (class name is coded in source) + "2014-01-07 14:12:54", returnType = TajoDataTypes.Type.TIMESTAMP, paramTypes = {@ParamTypes(paramTypes = {TajoDataTypes.Type.INT4}), @ParamTypes(paramTypes = {TajoDataTypes.Type.INT8})} ) TAJO-408 Improve function system

Functions & data types • supporting more functions and UDFs automatic function function registration function1 Tajo Master function2 Tajo Master runtime user defined user defined function3 function registration function @Description( functionName = "to_timestamp", description = "Convert UNIX epoch to time stamp", description registered at startup example = "> SELECT to_timestamp(1389071574);\n" (class name is coded in source) + "2014-01-07 14:12:54", returnType = TajoDataTypes.Type.TIMESTAMP, paramTypes = {@ParamTypes(paramTypes = {TajoDataTypes.Type.INT4}), @ParamTypes(paramTypes = {TajoDataTypes.Type.INT8})} ) TAJO-408 Improve function system

Functions & data types • supporting more functions and UDFs automatic function function registration function1 Tajo Master function2 Tajo Master runtime user defined user defined function3 function registration function @Description( functionName = "to_timestamp", description = "Convert UNIX epoch to time stamp", description registered at startup example = "> SELECT to_timestamp(1389071574);\n" (class name is coded in source) + "2014-01-07 14:12:54", returnType = TajoDataTypes.Type.TIMESTAMP, paramTypes = {@ParamTypes(paramTypes = {TajoDataTypes.Type.INT4}), @ParamTypes(paramTypes = {TajoDataTypes.Type.INT8})} TAJO-52 ) standard SQL TAJO-408 data types Improve function system

JDBC Driver, HCatalog TAJO-16, 433 Hive metastore TAJO-176 HCatalog JDBC Driver JDBC ANSI SQL SQL parser Tajo Algebra Query Master expression HiveQL HiveQL parser TAJO-101 HiveQL converter

Management TAJO-239 Improving Web UI

Management TAJO-564 Execution block progress

Management TAJO-589 Task progress

Management TAJO-468 Task detail info

Management TAJO-474 Task admin utility