Who am I? Takeshi NAKANO Senior Researcher / Architect. - - PowerPoint PPT Presentation
genn.ai () Realtime processing for everyone Takeshi Nakano @tf0054 1 Who am I? Takeshi NAKANO Senior Researcher / Architect. Co-authered Getting started with Solr in Japanese. Co-authered Hadoop Hacks in Japanese.
Realtime processing for everyone
1
Takeshi Nakano @tf0054
http://genn.ai/
Takeshi NAKANO
Senior Researcher / Architect. Co-authered Getting started with Solr in Japanese. Co-authered Hadoop Hacks in Japanese.
2
Writing “Getting started with Kafka” in Japanese.
http://genn.ai/
What is genn.ai?
Realtime processing Usecases
3
Let’s use genn.ai
Download & get your key Define a schema and a filter Make it real!
4
Customer A Customer B Customer C
Realtime Analysis Platform じゃらん Web Sites Client (Hotels) Marketer Analyst
http://genn.ai/
Gungnir Thrift Server Gungnir Rest Server Gungnir Cosole Storm nimbus Storm supervisor Gungnir Client
Kafka broker Storm supervisor Storm supervisor Kafka broker Kafka broker Gungnir Thrift Server
Gungnir Rest Server Gungnir Rest Server
MongoDB MetaStore Tuple Topic
Query
genn.ai UI
Query design view Query User account Tuple schema Topology JSON Tuple Tuple Tuple
Kafka MongoD B HDFS
User account Tuple schema
6
> cd gungnir-client > bin/gungnir -u tf0054 -p 2e2232073109 Picked up _JAVA_OPTIONS: -Dfile.encoding=UTF-8 2013/10/14 22:34:58 com.twitter.finagle.Init$ apply 情報: Finagle version 6.5.1 (rev=59..9cde41) built at 20130626-111057 Gungnir server connected... ec2-xxx.amazonaws.com/54.238.99.212:9290 Welcome tf0054 gungnir>
CREATE TUPLE userAction ( userId STRING, hotelId STRING ); CREATE TUPLE commitAction ( userId STRING, hotelId STRING, checkin TIMESTAMP('yyyy-MM-dd'), nights INT );
STRING Java type string TINYINT Java type byte SMALLINT Java type short INT Java type int BIGINT Java type long FLOAT Java type float DOUBLE Java type double TIMESTAMP long value LIST MAP STRUCT
FROM userAction AS ua, commitAction AS ca USING kafka_spout() EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user') ;
From EMIT
FROM userAction AS ua, commitAction AS ca USING kafka_spout() JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image USING mongo_fetch('getusers', 'userInfo') JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo USING mongo_fetch('hotelscom', 'hotel') EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user') ;
From JOIN
(user)
JOIN
(hotel)
EMIT
FROM userAction AS ua, commitAction AS ca USING kafka_spout() JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image USING mongo_fetch('getusers', 'userInfo') JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo USING mongo_fetch('hotelscom', 'hotel') INTO stream_0 ; FROM stream_0 EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user') ;
From JOIN
(user)
JOIN
(hotel)
stream_0
EMIT
(user)
FROM userAction AS ua, commitAction AS ca USING kafka_spout() JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image USING mongo_fetch('getusers', 'userInfo') JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo USING mongo_fetch('hotelscom', 'hotel') INTO stream_0 ; FROM stream_0 EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user') ; FROM stream_0 BEGIN GROUP BY hotelId EACH count() AS count, * EMIT hotelId, hotelname, photo, count USING kafka_emit('${TOPOLOGY_ID}_hotel') END GROUP ;
From JOIN JOIN EMIT
(user)
stream_0
GROUP BY EMIT
(hotel)
EACH sum(nights) EACH avg(nights)
SUBMIT TOPOLOGY; DESC TOPOLOGY; SHOW TOPOLOGIES; STOP TOPOLOGY xxx; DROP TOPOLOGY xxx;
gungnir> SUBMIT TOPOLOGY; OK gungnir> DESC TOPOLOGY; { "id":"52373363e4b0a525adadb4f2", "explain":" ... ", "status":"RUNNING", "owner":"tf0054@gmail.com", "createTime":"2013-09-16T16:35:47.432Z", "summary":{ ... } } gungnir> STOP TOPOLOGY 52373363e4b0a525adadb4f2 ; gungnir> DELETE TOPOLOGY 52373363e4b0a525adadb4f2 ;
TRACK userAction {"userId":"siniida","hotelId":"226979"} TRACK commitAction {"userId":"siniida","hotelId":"226979","checkin":"2013-10-18","nights":1}
> bin/kafka-consumer.sh 52373363e4b0a525adadb4f2_user ;
> bin/kafka-consumer.sh 52373363e4b0a525adadb4f2_hotel; gungnir> DESC TOPOLOGY; { "id":"52373363e4b0a525adadb4f2", "explain":" ... ", "status":"RUNNING", "owner":"tf0054@gmail.com", ...
... USING kafka_emit('${TOPOLOGY_ID}_user')
> curl -v -H "Content-type: application/json" -X POST
http://api.genn.ai/gungnir/v1.0/track/525e67f8e4b0e7cfca98d040/userAction gungnir> DESC USER; { "id":"525e67f8e4b0e7cfca98d040", "name":"siniida", "password":"CmGOdN2UU/Xtmk0wRCZKzDy/BN65LAHPCp2+1KMDOOM=", "createTime":"2013-10-16T10:18:32.849Z" } gungnir>
Built on Storm
Converts properly (e.g. building AST) The Good Parts (e.g. Supporting parallelism setting) ..
genn.ai wants friends!
genn.ai codes will be open-sourced in the future. We are looking for co-challenging partners for
improving genn.ai.
You can use genn.ai on your web sites completely for
free (infrastructure etc..) if we can share the dream together.
Please feel free to contact us!
http://genn.ai/ tf0054@gmail.com
http://genn.ai/
19
20