Delivering Real-Time Data with Azure DevUp October 16, 2019 Chad - - PowerPoint PPT Presentation

▶
delivering real time data with azure
SMART_READER_LITE
LIVE PREVIEW

Delivering Real-Time Data with Azure DevUp October 16, 2019 Chad - - PowerPoint PPT Presentation

Sep 27, 2022 232 likes •866 views

Delivering Real-Time Data with Azure DevUp October 16, 2019 Chad Green @ChadGreen Delivering Real-Time Data with Azure 1 Platinum Sponsors Lanyards Sponsor Celebration Sponsor Silver Sponsors Notebook Sponsor Registration Sponsor Chad

slide-1
SLIDE 1

1

@ChadGreen Delivering Real-Time Data with Azure

Delivering Real-Time Data with Azure

DevUp

Chad Green

October 16, 2019

slide-2
SLIDE 2

Silver Sponsors

Platinum Sponsors

Celebration Sponsor Lanyards Sponsor Notebook Sponsor Registration Sponsor

slide-3
SLIDE 3

3

@ChadGreen Delivering Real-Time Data with Azure

u ) 2 #

chadgreen@chadgreen.com

Chad Green

Director of Software Development ScholarRx

chadwickegreen ChadGreen ChadGreen.com

slide-4
SLIDE 4

4

@ChadGreen Delivering Real-Time Data with Azure

Agenda

Delivering Real-Time Data with Azure

  • Rea

Real-Time ime Data Data and and Azur ure

  • Consuming
  • nsuming Data

Data Thr Throug

  • ugh

h Eve Event nt Hubs ubs

  • Analyz

nalyzing ing Data Data with ith Str tream m Analytics nalytics

  • Ing

ngestin ting g Str treaming Data ming Data into into Power B BI

  • Building

uilding Real al-Ti Time me Visualiz Visualizations ations with ith Power BI

slide-5
SLIDE 5

5

@ChadGreen Delivering Real-Time Data with Azure

Real-Time Data and Azure

Delivering Real-Time Data with Azure

slide-6
SLIDE 6

6

@ChadGreen Delivering Real-Time Data with Azure

Real-Time Processing

Real-Time Data and Azure

Da Data So Source ces Da Data S Stor

  • rage

Ingest estion ion Stream P Process essin ing Analyt lytic ical l Da Data Sto Store Analyt lytic ics s and Report

  • rtin

ing Batch Process essin ing Orchest stration ion

slide-7
SLIDE 7

7

@ChadGreen Delivering Real-Time Data with Azure

  • Ing

ngest, pr t, proce

  • cess, and

ss, and stor store mess ssage ges in in real al-time ime

  • Cannot

annot block block the the ing ingestion pipe stion pipeline line

  • Act

ct on data quickly

  • n data quickly

Challenges

Real-Time Data and Azure

slide-8
SLIDE 8

8

@ChadGreen Delivering Real-Time Data with Azure

Architecture

Real-Time Data and Azure

Da Data So Source ces Da Data S Stor

  • rage

Ingest estion ion Stream P Process essin ing Analyt lytic ical l Da Data Sto Store Analyt lytic ics s and Report

  • rtin

ing Batch Process essin ing Orchest stration ion

slide-9
SLIDE 9

9

@ChadGreen Delivering Real-Time Data with Azure

Technology Choices

Real-Time Data and Azure

Da Data So Source ces Da Data S Stor

  • rage

Ingest estion ion Stream P Process essin ing Analyt lytic ical l Da Data Sto Store Analyt lytic ics s and Report

  • rtin

ing Batch Process essin ing Orchest stration ion Ingest estion ion Azure E Eve vent Hu Hubs Azure I IoT

  • T Hu

Hub Apache K Kafka Da Data S Stor

  • rage

Azure S Stor

  • rage B

Blob lobs Azure Da Data La Lake S e Stor

  • re

Stream P Process essin ing

Az Azur ure Str tream Ana Analy lytics

Sto Storm Sp Spark Str Streaming Analyt lytic ical l Da Data Sto Store SQL DW DW Hb Hbase Spark rk Hiv Hive

Cosm

  • smos D
  • s DB

Analyt lytic ics s and Report

  • rtin

ing Azure re Analys lysis is Servic rvices Power B er BI Microsof

  • soft

Ex Exce cel

slide-10
SLIDE 10

10

@ChadGreen Delivering Real-Time Data with Azure

Technology Choices

Real-Time Data and Azure

Real-Tim ime M e Messa ssage I e Ingest estion ion Stream P Process essin ing Analyt lytic ics a s and Repor

  • rtin

ing Da Data S Stor

  • rage

Analytical Da Data S Stor

  • re

Eve vent Hu Hubs Stream A Analyt lytic ics Power B er BI Stor

  • rage B

Blob lob C Con

  • ntainer

Cosm

  • smos DB

DB

slide-11
SLIDE 11

11

@ChadGreen Delivering Real-Time Data with Azure

Demo Scenario

Real-Time Data and Azure

  • Use

ser answ answers pr practice actice te test st que question stion

  • Syste

ystem m recor cords ds inte interact action ion

  • Dashboar

Dashboard d sho showing ing real al-ti time me usag usage of

  • f te

test st functionality unctionality

slide-12
SLIDE 12

12

@ChadGreen Delivering Real-Time Data with Azure

Demo Scenario

Real-Time Data and Azure

Cl Client nt Applica lication tion

slide-13
SLIDE 13

13

@ChadGreen Delivering Real-Time Data with Azure

Demo Scenario

Real-Time Data and Azure

Cl Client nt Applica lication tion

slide-14
SLIDE 14

14

@ChadGreen Delivering Real-Time Data with Azure

Demo Scenario

Real-Time Data and Azure

Cl Client nt Applica lication tion

slide-15
SLIDE 15

15

@ChadGreen Delivering Real-Time Data with Azure

Demo Scenario

Real-Time Data and Azure

Cl Client nt Applica lication tion

slide-16
SLIDE 16

16

@ChadGreen Delivering Real-Time Data with Azure

Demo Scenario

Real-Time Data and Azure

Cl Client nt Applica lication tion

slide-17
SLIDE 17

17

@ChadGreen Delivering Real-Time Data with Azure

Demo Scenario

Real-Time Data and Azure

Cl Client nt Applica lication tion

slide-18
SLIDE 18

18

@ChadGreen Delivering Real-Time Data with Azure

Consuming Data Through Event Hubs

Delivering Real-Time Data with Azure

slide-19
SLIDE 19

19

@ChadGreen Delivering Real-Time Data with Azure

Azure Event Hubs

Simple, secure, and scalable real-time data ingestion

Fully ully manage anaged, r d, real al-ti time me data data ing ingest stion ion se service vice that that is is sim simple ple, tr , truste usted, d, and and scalable scalable.

slide-20
SLIDE 20

20

@ChadGreen Delivering Real-Time Data with Azure

Si Simp mple Sec Secure Scalab alable le Open en

Azure Event Hubs

Simple, secure, and scalable real-time data ingestion

slide-21
SLIDE 21

21

@ChadGreen Delivering Real-Time Data with Azure

Event nt Produce ucer Eve vent In Inges gestion

  • n

Even ent Co Consumer er Even ent Co Consumer er Even ent Co Consumer er

Event-Driven Architecture

Simple, secure, and scalable real-time data ingestion

slide-22
SLIDE 22

22

@ChadGreen Delivering Real-Time Data with Azure

Throughput Units

Azure Event Hubs

Variable reserved capacities assigned to the Event Hub namespace Ingres ess 1-Mb or 1,000 events Eg Egress ss 2-Mb or 4,096 events Up to 20 TUs via Portal Up to 40 TUs via Support Ticket

slide-23
SLIDE 23

23

@ChadGreen Delivering Real-Time Data with Azure

Provisioning

Consuming Data Through Event Hubs

De Demo Provision vision Azur ure Eve Event nt Hubs ubs

slide-24
SLIDE 24

24

@ChadGreen Delivering Real-Time Data with Azure

Sending Data to Event Hubs

Consuming Data Through Event Hubs

De Demo Sending nding Data Data to Eve to Even H n Hubs ubs

slide-25
SLIDE 25

25

@ChadGreen Delivering Real-Time Data with Azure

Analyzing Data with Stream Analytics

Delivering Real-Time Data with Azure

slide-26
SLIDE 26

26

@ChadGreen Delivering Real-Time Data with Azure

Azure Stream Analytics

Serverless real-time analytics

De Delive liver po powerful ul insig insights hts from

  • m

your your str streamin aming g data data with ith ease ase, in , in real al tim time.

slide-27
SLIDE 27

27

@ChadGreen Delivering Real-Time Data with Azure

Potential Use Cases

Azure Stream Analytics

  • Analyz

nalyze real al-tim ime te tele lemetry str streams from

  • m

Io IoT device vices

  • Web logs/

gs/clic ickstr tream anal am analyt ytic ics

  • Ge

Geosp spatial tial anal analytic ytics s for fleet t manage manageme ment and and drive iverle less ss vehic hicle les

  • Re

Remo mote mo moni nito toring and ing and predic ictiv tive mai maint ntena nance ce of hi high gh val alue as asse sets ts

  • Re

Real al-time time anal analytic ytics s on n Point nt of Sal ale dat ata a for inv nventor

  • ry contr

ntrol l and and ano anomaly maly detectio tion

slide-28
SLIDE 28

28

@ChadGreen Delivering Real-Time Data with Azure

How does Stream Analytics work

Azure Stream Analytics

slide-29
SLIDE 29

29

@ChadGreen Delivering Real-Time Data with Azure

Key Capabilities and Benefits

Azure Stream Analytics

Ea Ease se of

  • f

Gettin ing g Sta Started Progr gramme mmer r Producti tivi vity ty Fully Manag anaged Low T w Total al Cos Cost of

  • f

Owners rship ip (TCO) Reliabilit lity Performa rmance ce

slide-30
SLIDE 30

30

@ChadGreen Delivering Real-Time Data with Azure

Key Capabilities and Benefits

Azure Stream Analytics

Ea Ease se of

  • f

Gettin ing g Sta Started Progr gramme mmer r Producti tivi vity ty Fully Manag anaged Low T w Total al Cos Cost of

  • f

Owners rship ip (TCO) Reliabilit lity Performa rmance ce

slide-31
SLIDE 31

31

@ChadGreen Delivering Real-Time Data with Azure

Key Capabilities and Benefits

Azure Stream Analytics

Ea Ease se of

  • f

Gettin ing g Sta Started Progr gramme mmer r Producti tivi vity ty Fully Manag anaged Low T w Total al Cos Cost of

  • f

Owners rship ip (TCO) Reliabilit lity Performa rmance ce

slide-32
SLIDE 32

32

@ChadGreen Delivering Real-Time Data with Azure

Key Capabilities and Benefits

Azure Stream Analytics

Ea Ease se of

  • f

Gettin ing g Sta Started Progr gramme mmer r Producti tivi vity ty Fully Manag anaged Low T w Total al Cos Cost of

  • f

Owners rship ip (TCO) Reliabilit lity Performa rmance ce

slide-33
SLIDE 33

33

@ChadGreen Delivering Real-Time Data with Azure

Key Capabilities and Benefits

Azure Stream Analytics

Ea Ease se of

  • f

Gettin ing g Sta Started Progr gramme mmer r Producti tivi vity ty Fully Manag anaged Low T w Total al Cos Cost of

  • f

Owners rship ip (TCO) Reliabilit lity Performa rmance ce

slide-34
SLIDE 34

34

@ChadGreen Delivering Real-Time Data with Azure

Key Capabilities and Benefits

Azure Stream Analytics

Ea Ease se of

  • f

Gettin ing g Sta Started Progr gramme mmer r Producti tivi vity ty Fully Manag anaged Low T w Total al Cos Cost of

  • f

Owners rship ip (TCO) Reliabilit lity Performa rmance ce

slide-35
SLIDE 35

35

@ChadGreen Delivering Real-Time Data with Azure

Key Capabilities and Benefits

Azure Stream Analytics

Ea Ease se of

  • f

Gettin ing g Sta Started Progr gramme mmer r Producti tivi vity ty Fully Manag anaged Low T w Total al Cos Cost of

  • f

Owners rship ip (TCO) Reliabilit lity Performa rmance ce

slide-36
SLIDE 36

36

@ChadGreen Delivering Real-Time Data with Azure

Provisioning

Analyzing Data with Stream Analytics

De Demo Creat ate a a Str tream m Analytics nalytics Job

  • b
slide-37
SLIDE 37

37

@ChadGreen Delivering Real-Time Data with Azure

Stream Analytics Query Language

Analyzing Data with Stream Analytics

  • Aggregate
  • Analytic
  • Array
  • GeoSpatial
  • Input Metadata
  • Record
  • Windowing
  • Scalar
slide-38
SLIDE 38

38

@ChadGreen Delivering Real-Time Data with Azure

Stream Analytics Query Language

Analyzing Data with Stream Analytics

Eve Events nts and and Tim Time SELECT * FROM SensorReadings TIMESTAMP BY time SELECT System.Timestamp AS Time FROM SensorReadings

slide-39
SLIDE 39

39

@ChadGreen Delivering Real-Time Data with Azure

Stream Analytics Query Language – Windowing

Analyzing Data with Stream Analytics

SELECT sensorId, COUNT(*) AS Count FROM SensorReadings TIMESTAMP BY time GROUP BY sensorId, TumblingWindow(second, 10)

slide-40
SLIDE 40

40

@ChadGreen Delivering Real-Time Data with Azure

Stream Analytics Query Language – Windowing

Analyzing Data with Stream Analytics

SELECT sensorId, COUNT(*) AS Count, AVG(temp) FROM SensorReadings TIMESTAMP BY time GROUP BY sensorId, HoppingWindow(second, 10 , 5)

slide-41
SLIDE 41

41

@ChadGreen Delivering Real-Time Data with Azure

Stream Analytics Query Language – Windowing

Analyzing Data with Stream Analytics

SELECT sensorId, MIN(temp) as temp FROM SensorReadings TIMESTAMP BY time GROUP BY sensorId, SlidingWindow(second, 5) HAVING MIN(temp) > 75

slide-42
SLIDE 42

42

@ChadGreen Delivering Real-Time Data with Azure

Stream Analytics Query Language – Joining Multiple Steams

Analyzing Data with Stream Analytics

SELECT s1.time, s1.dspl, s1.hmdt as previousHmdt, s2.hmdt as newHmdt, datediff(ss, s1.time, s2.time) as secondsApart FROM SensorData s1 timestamp by time JOIN SensorData s2 timestamp by time ON s1.dspl = s2.dspl AND DATEDIFF(s, s1, s2) BETWEEN 0 AND 5 WHERE (s2.hmdt – s1.hmdt >= .1) or (s1.hmdt – s2.hmdt >= .1)

slide-43
SLIDE 43

43

@ChadGreen Delivering Real-Time Data with Azure

Stream Analytics Query Language – Reference Data JOIN

Analyzing Data with Stream Analytics

SELECT SensorReadings.sensorID, SensorReadings.temp FROM SensorReadings JOIN thresholdRefData ON SensorReadings.sensorID = thresholdRefData.sensorID WHERE SensorReadings.temp > thresholdRefData.value

slide-44
SLIDE 44

44

@ChadGreen Delivering Real-Time Data with Azure

Stream Analytics Query Language – Multiple Outputs

Analyzing Data with Stream Analytics

SELECT * INTO outputLog FROM SensorReadings SELECT * INTO outputTempAlert FROM SensorReadings WHERE temp > 75

slide-45
SLIDE 45

45

@ChadGreen Delivering Real-Time Data with Azure

Querying

Analyzing Data with Stream Analytics

De Demo Que Query y a a Str tream

slide-46
SLIDE 46

46

@ChadGreen Delivering Real-Time Data with Azure

Ingesting Streaming Data into Power BI

Delivering Real-Time Data with Azure

slide-47
SLIDE 47

47

@ChadGreen Delivering Real-Time Data with Azure

Microsoft Power BI

Ingesting Streaming Data into Power BI

Provide vides s inte interact active ive visualiz visualizations ations and and busine business ss inte intellig lligence nce capabilitie capabilities s with ith an an inte interface ace sim simple ple enoug nough h for

  • r e

end use nd users s to cr to create ate the their ir

  • wn

n repor ports ts and and dashboar dashboards. ds.

slide-48
SLIDE 48

48

@ChadGreen Delivering Real-Time Data with Azure

Microsoft Power BI

Ingesting Streaming Data into Power BI

Cre reate

Co Collabor borate te and and Shar hare Access Ins s Insights hts from m Anywhe nywhere

slide-49
SLIDE 49

49

@ChadGreen Delivering Real-Time Data with Azure

Microsoft Power BI

Ingesting Streaming Data into Power BI

Cre reate

Co Collabor borate te and and Shar hare Access Ins s Insights hts from m Anywhe nywhere

slide-50
SLIDE 50

50

@ChadGreen Delivering Real-Time Data with Azure

Microsoft Power BI

Ingesting Streaming Data into Power BI

Cre reate

Co Collabor borate te and and Shar hare Access Ins s Insights hts from m Anywhe nywhere

slide-51
SLIDE 51

51

@ChadGreen Delivering Real-Time Data with Azure

Why Microsoft Power BI

Ingesting Streaming Data into Power BI

slide-52
SLIDE 52

52

@ChadGreen Delivering Real-Time Data with Azure

Why Microsoft Power BI

Ingesting Streaming Data into Power BI

slide-53
SLIDE 53

53

@ChadGreen Delivering Real-Time Data with Azure

Why Microsoft Power BI

Ingesting Streaming Data into Power BI

slide-54
SLIDE 54

54

@ChadGreen Delivering Real-Time Data with Azure

Why Microsoft Power BI

Ingesting Streaming Data into Power BI

slide-55
SLIDE 55

55

@ChadGreen Delivering Real-Time Data with Azure

Why Microsoft Power BI

Ingesting Streaming Data into Power BI

slide-56
SLIDE 56

56

@ChadGreen Delivering Real-Time Data with Azure

Why Microsoft Power BI

Ingesting Streaming Data into Power BI

slide-57
SLIDE 57

57

@ChadGreen Delivering Real-Time Data with Azure

Why Microsoft Power BI

Ingesting Streaming Data into Power BI

slide-58
SLIDE 58

58

@ChadGreen Delivering Real-Time Data with Azure

Why Microsoft Power BI

Ingesting Streaming Data into Power BI

slide-59
SLIDE 59

59

@ChadGreen Delivering Real-Time Data with Azure

Power BI Real-Time Datasets

Ingesting Streaming Data into Power BI

  • Support very fast dashboard tiles
  • Limited graphic options
  • Uses a Redis cache under the covers
  • Keeps approximately one hour of data
slide-60
SLIDE 60

60

@ChadGreen Delivering Real-Time Data with Azure

Real-Time Datasets

Ingesting Streaming Data into Power BI

De Demo Creat ating ing a a Real al-Time ime Datase Dataset t with ith Str tream m Analytics nalytics

slide-61
SLIDE 61

61

@ChadGreen Delivering Real-Time Data with Azure

Real-Time Datasets

Ingesting Streaming Data into Power BI

De Demo Power BI S Str treaming ming Title Titles

slide-62
SLIDE 62

62

@ChadGreen Delivering Real-Time Data with Azure

Real-Time Datasets

Ingesting Streaming Data into Power BI

De Demo Rea Real-Time ime Custo ustom m Repor port t Visualiz Visualizations ations

slide-63
SLIDE 63

63

@ChadGreen Delivering Real-Time Data with Azure

u ) 2 #

chadgreen@chadgreen.com

Chad Green

Director of Software Development ScholarRx

chadwickegreen ChadGreen ChadGreen.com