Who am I? Takeshi NAKANO Senior Researcher / Architect. - PowerPoint PPT Presentation

genn.ai (源内) Realtime processing for everyone Takeshi Nakano @tf0054 1

Who am I?  Takeshi NAKANO  Senior Researcher / Architect.  Co-authered Getting started with Solr in Japanese.  Co-authered Hadoop Hacks in Japanese.  Writing “Getting started with Kafka” in Japanese. http://genn.ai/ 2

Agenda  What is genn.ai?  Let’s use genn.ai  Realtime processing  Download & get your key  Usecases  Define a schema and a filter  Make it real! http://genn.ai/ 3

じゃらん A usecase of genn.ai Customer A Web Sites Customer B Customer C Marketer Analyst Realtime Analysis Platform Client (Hotels) 4

Visual editor for genn.ai

Structure of genn.ai MongoDB User account MetaStore Tuple schema genn.ai UI Query design view Gungnir Gungnir Storm Client Query Thrift Server Gungnir nimbus Query Topology on finagle on finagle Thrift Server Gungnir Cosole User account Kafka Tuple schema MongoD Gungnir B Kafka Storm Rest Server Gungnir Kafka Storm broker supervisor Tuple Tuple Gungnir Kafka Storm on finagle JSON Tuple Rest Server broker supervisor Rest Server broker supervisor HDFS Tuple Topic http://genn.ai/ 6

http://dev.genn.ai/ Preparing for starting.

http://dev.genn.ai/ Preparing for starting. > cd gungnir - client > bin / gungnir - u tf0054 - p 2e2232073109 Picked up _ JAVA _ OPTIONS : - Dfile . encoding = UTF - 8 2013 / 10 / 14 22 : 34 : 58 com . twitter . finagle . Init $ apply 情報 : Finagle version 6 . 5 . 1 ( rev = 59 .. 9cde41 ) built at 20130626 - 111057 Gungnir server connected ... ec2 - xxx . amazonaws . com / 54 . 238 . 99 . 212 : 9290 Welcome tf0054 gungnir >

Define your schema! CREATE TUPLE userAction ( STRING Java type string userId STRING, hotelId STRING TINYINT Java type byte ); SMALLINT Java type short CREATE TUPLE commitAction ( INT Java type int userId STRING, hotelId STRING, BIGINT Java type long checkin TIMESTAMP('yyyy-MM-dd'), FLOAT Java type float nights INT DOUBLE Java type double ); TIMESTAMP long value LIST MAP STRUCT

Define your filter! (1/4) FROM userAction AS ua, commitAction AS ca USING kafka_spout() EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user') ; From EMIT

Define your filter! (2/4) FROM userAction AS ua, commitAction AS ca USING kafka_spout() JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image USING mongo_fetch('getusers', 'userInfo') JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo USING mongo_fetch('hotelscom', 'hotel') EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user') ; JOIN JOIN From EMIT (user) (hotel)

Define your filter! (3/4) FROM userAction AS ua, commitAction AS ca USING kafka_spout() JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image USING mongo_fetch('getusers', 'userInfo') JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo USING mongo_fetch('hotelscom', 'hotel') INTO stream_0 ; FROM stream_0 EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user') ; JOIN JOIN EMIT From (user) (user) (hotel) stream_0

Define your filter! (4/4) FROM userAction AS ua, commitAction AS ca USING kafka_spout() JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image USING mongo_fetch('getusers', 'userInfo') JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo USING mongo_fetch('hotelscom', 'hotel') INTO stream_0 ; FROM stream_0 EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user') ; FROM stream_0 BEGIN GROUP BY hotelId EACH count() AS count, * EMIT hotelId, hotelname, photo, count USING kafka_emit('${TOPOLOGY_ID}_hotel') END GROUP ; EACH sum ( nights ) EMIT EACH avg ( nights ) (user) From JOIN JOIN stream_0 GROUP EMIT BY (hotel)

Register your filter! and ... SUBMIT TOPOLOGY; gungnir > SUBMIT TOPOLOGY; OK gungnir > DESC TOPOLOGY; { DESC TOPOLOGY; " id ":" 52373363e4b0a525adadb4f2 " , " explain ":" ... " , " status ":" RUNNING " , " owner ":" tf0054@gmail . com " , SHOW TOPOLOGIES; " createTime ":" 2013 - 09 - 16T16 : 35 : 47 . 432Z " , " summary ":{ ... } STOP TOPOLOGY xxx; } gungnir > STOP TOPOLOGY 52373363e4b0a525adadb4f2 ; DROP TOPOLOGY xxx; gungnir > DELETE TOPOLOGY 52373363e4b0a525adadb4f2 ;

Testing it! TRACK userAction {"userId":"siniida","hotelId":"226979"} TRACK commitAction {"userId":"siniida","hotelId":"226979","checkin":"2013-10-18","nights":1} gungnir > DESC TOPOLOGY; { " id ":" 52373363e4b0a525adadb4f2 " , " explain ":" ... " , " status ":" RUNNING " , " owner ":" tf0054@gmail . com " , ... ... USING kafka_emit('${TOPOLOGY_ID}_user') > bin / kafka - consumer . sh 52373363e4b0a525adadb4f2 _ user ; or > bin / kafka - consumer . sh 52373363e4b0a525adadb4f2 _ hotel;

Knowing http tuple receiver gungnir > DESC USER; { " id ":" 525e67f8e4b0e7cfca98d040 " , " name ":" siniida " , " password ":" CmGOdN2UU / Xtmk0wRCZKzDy / BN65LAHPCp2 + 1KMDOOM =" , " createTime ":" 2013 - 10 - 16T10 : 18 : 32 . 849Z " } gungnir > > curl - v - H " Content - type : application / json " - X POST - d '{" userId ":" tf0054 " , " hotelId ":" 226979 "}' http :// api . genn . ai / gungnir / v1 . 0 / track / 525e67f8e4b0e7cfca98d040 / userAction

Design of genn.ai  Built on Storm  Converts properly (e.g. building AST)  The Good Parts (e.g. Supporting parallelism setting)  ..

improving genn.ai. free (infrastructure etc..) if we can share the dream together. The future of genn.ai  genn.ai wants friends!  genn.ai codes will be open-sourced in the future.  We are looking for co-challenging partners for  You can use genn.ai on your web sites completely for  Please feel free to contact us!  http://genn.ai/  tf0054@gmail.com

Thanks! http://genn.ai/ 19

20