Delivering Real-Time Data with Azure DevUp October 16, 2019 Chad Green @ChadGreen Delivering Real-Time Data with Azure 1
Platinum Sponsors Lanyards Sponsor Celebration Sponsor Silver Sponsors Notebook Sponsor Registration Sponsor
Chad Green Director of Software Development ScholarRx chadgreen@chadgreen.com u chadwickegreen ) ChadGreen 2 ChadGreen.com # @ChadGreen Delivering Real-Time Data with Azure 3
Agenda Delivering Real-Time Data with Azure • Rea Real-Time ime Data Data and and Azur ure • Consuming onsuming Data Data Thr Throug ough h Eve Event nt Hubs ubs • Analyz nalyzing ing Data Data with ith Str tream m Analytics nalytics • Ing ngestin ting g Str treaming Data ming Data into into Power B BI • Building uilding Real al-Ti Time me Visualiz Visualizations ations with ith Power BI @ChadGreen Delivering Real-Time Data with Azure 4
Real-Time Data and Azure Delivering Real-Time Data with Azure @ChadGreen Delivering Real-Time Data with Azure 5
Real-Time Processing Real-Time Data and Azure Batch Data S Da Stor orage Process essin ing Analyt lytic ical l Analyt lytic ics s Data Da Da Data and Sto Store So Source ces Report ortin ing Ingest estion ion Stream P Process essin ing Orchest stration ion @ChadGreen Delivering Real-Time Data with Azure 6
Challenges Real-Time Data and Azure • Ing ngest, pr t, proce ocess, and ss, and stor store mess ssage ges in in real al-time ime • Cannot annot block block the the ing ingestion pipe stion pipeline line • Act ct on data quickly on data quickly @ChadGreen Delivering Real-Time Data with Azure 7
Architecture Real-Time Data and Azure Batch Data S Da Stor orage Process essin ing Analyt lytic ical l Analyt lytic ics s Data Da Da Data and Sto Store So Source ces Report ortin ing Ingest estion ion Stream P Process essin ing Orchest stration ion @ChadGreen Delivering Real-Time Data with Azure 8
Technology Choices Real-Time Data and Azure Da Data S Stor orage Batch Da Data S Stor orage Analyt lytic ics s Azure S Stor orage B Blob lobs Process essin ing Analyt lytic ical l and Azure Da Data La Lake S e Stor ore Data Da Report ortin ing Sto Store Analyt lytic ical l Azure re Analyt lytic ics s Data Da SQL DW DW Analys lysis is Data Da and Sto Store Hb Hbase Servic rvices Source So ces Ingest estion ion Stream P Process essin ing Report ortin ing Spark rk Hiv Hive Power B er BI Azure E Eve vent Hu Hubs Ingest estion ion Stream P Process essin ing Az Azur ure Str tream Ana Analy lytics Cosm osmos D os DB Storm Sto Azure I IoT oT Hu Hub Microsof osoft Sp Spark Str Streaming Apache K Kafka Exce Ex cel Orchest stration ion @ChadGreen Delivering Real-Time Data with Azure 9
Technology Choices Real-Time Data and Azure Real-Tim ime M e Messa ssage I e Ingest estion ion Stream P Process essin ing Analyt lytic ics a s and Repor ortin ing Eve vent Hu Hubs Stream A Analyt lytic ics Power B er BI Data S Da Stor orage Analytical Da Data S Stor ore Stor orage B Blob lob C Con ontainer Cosm osmos DB DB @ChadGreen Delivering Real-Time Data with Azure 10
Demo Scenario Real-Time Data and Azure • Use ser answ answers pr practice actice te test st que question stion • Syste ystem m recor cords ds inte interact action ion • Dashboar Dashboard d sho showing ing real al-ti time me usag usage of of te test st functionality unctionality @ChadGreen Delivering Real-Time Data with Azure 11
Demo Scenario Real-Time Data and Azure lication tion nt Applica Client Cl @ChadGreen Delivering Real-Time Data with Azure 12
Demo Scenario Real-Time Data and Azure lication tion nt Applica Client Cl @ChadGreen Delivering Real-Time Data with Azure 13
Demo Scenario Real-Time Data and Azure lication tion nt Applica Client Cl @ChadGreen Delivering Real-Time Data with Azure 14
Demo Scenario Real-Time Data and Azure lication tion nt Applica Client Cl @ChadGreen Delivering Real-Time Data with Azure 15
Demo Scenario Real-Time Data and Azure lication tion nt Applica Client Cl @ChadGreen Delivering Real-Time Data with Azure 16
Demo Scenario Real-Time Data and Azure lication tion nt Applica Client Cl @ChadGreen Delivering Real-Time Data with Azure 17
Consuming Data Through Event Hubs Delivering Real-Time Data with Azure @ChadGreen Delivering Real-Time Data with Azure 18
Azure Event Hubs Simple, secure, and scalable real-time data ingestion Fully ully manage anaged, r d, real al-ti time me data data ing ingest stion ion se service vice that that is is sim simple ple, tr , truste usted, d, and and scalable scalable. @ChadGreen Delivering Real-Time Data with Azure 19
Azure Event Hubs Simple, secure, and scalable real-time data ingestion Si Simp mple Sec Secure Scalab alable le Open en @ChadGreen Delivering Real-Time Data with Azure 20
Event-Driven Architecture Simple, secure, and scalable real-time data ingestion Even ent Co Consumer er Event nt Produce ucer Eve vent In Inges gestion on Even ent Co Consumer er Even ent Co Consumer er @ChadGreen Delivering Real-Time Data with Azure 21
Throughput Units Azure Event Hubs Variable reserved capacities assigned to the Event Hub namespace Ingres ess Eg Egress ss 1-Mb or 1,000 events 2-Mb or 4,096 events Up to 20 TUs via Portal Up to 40 TUs via Support Ticket @ChadGreen Delivering Real-Time Data with Azure 22
Provisioning Consuming Data Through Event Hubs De Demo Provision vision Azur ure Eve Event nt Hubs ubs @ChadGreen Delivering Real-Time Data with Azure 23
Sending Data to Event Hubs Consuming Data Through Event Hubs De Demo Sending nding Data Data to Eve to Even H n Hubs ubs @ChadGreen Delivering Real-Time Data with Azure 24
Analyzing Data with Stream Analytics Delivering Real-Time Data with Azure @ChadGreen Delivering Real-Time Data with Azure 25
Azure Stream Analytics Serverless real-time analytics De Delive liver po powerful ul insig insights hts from om your your str streamin aming g data data with ith ease ase, in , in real al tim time. @ChadGreen Delivering Real-Time Data with Azure 26
Potential Use Cases Azure Stream Analytics Analyz nalyze real al-tim ime te tele lemetry str streams from om • IoT device Io vices Web logs/ gs/clic ickstr tream anal am analyt ytic ics • Geosp Ge spatial tial anal analytic ytics s for fleet t manage manageme ment • and and drive iverle less ss vehic hicle les Re Remo mote mo moni nito toring and ing and predic ictiv tive • mai maint ntena nance ce of hi high gh val alue as asse sets ts Re Real al-time time anal analytic ytics s on n Point nt of Sal ale dat ata a for • inv nventor ory contr ntrol l and and ano anomaly maly detectio tion @ChadGreen Delivering Real-Time Data with Azure 27
How does Stream Analytics work Azure Stream Analytics @ChadGreen Delivering Real-Time Data with Azure 28
Key Capabilities and Benefits Azure Stream Analytics Low T w Total al Ea Ease se of of Progr gramme mmer r Fully Cos Cost of of Gettin ing g Reliabilit lity Performa rmance ce Producti tivi vity ty Manag anaged Owners rship ip Started Sta (TCO) @ChadGreen Delivering Real-Time Data with Azure 29
Key Capabilities and Benefits Azure Stream Analytics Low T w Total al Ea Ease se of of Progr gramme mmer r Fully Cos Cost of of Gettin ing g Reliabilit lity Performa rmance ce Producti tivi vity ty Manag anaged Owners rship ip Started Sta (TCO) @ChadGreen Delivering Real-Time Data with Azure 30
Key Capabilities and Benefits Azure Stream Analytics Low T w Total al Ea Ease se of of Progr gramme mmer r Fully Cos Cost of of Gettin ing g Reliabilit lity Performa rmance ce Producti tivi vity ty Manag anaged Owners rship ip Started Sta (TCO) @ChadGreen Delivering Real-Time Data with Azure 31
Key Capabilities and Benefits Azure Stream Analytics Low T w Total al Ea Ease se of of Progr gramme mmer r Fully Cos Cost of of Gettin ing g Reliabilit lity Performa rmance ce Producti tivi vity ty Manag anaged Owners rship ip Started Sta (TCO) @ChadGreen Delivering Real-Time Data with Azure 32
Key Capabilities and Benefits Azure Stream Analytics Low T w Total al Ea Ease se of of Progr gramme mmer r Fully Cos Cost of of Gettin ing g Reliabilit lity Performa rmance ce Producti tivi vity ty Manag anaged Owners rship ip Started Sta (TCO) @ChadGreen Delivering Real-Time Data with Azure 33
Key Capabilities and Benefits Azure Stream Analytics Low T w Total al Ea Ease se of of Progr gramme mmer r Fully Cos Cost of of Gettin ing g Reliabilit lity Performa rmance ce Producti tivi vity ty Manag anaged Owners rship ip Started Sta (TCO) @ChadGreen Delivering Real-Time Data with Azure 34
Recommend
More recommend
