SLIDE 1
1
@ChadGreen Delivering Real-Time Data with Azure
Delivering Real-Time Data with Azure
South Florida Software Developer’s Conference
Chad Green
February 29, 2020
2
@ChadGreen Delivering Real-Time Data with Azure
Delivering Real-Time Data with Azure South Florida Software - - PowerPoint PPT Presentation
Delivering Real-Time Data with Azure South Florida Software - - PowerPoint PPT Presentation
Delivering Real-Time Data with Azure South Florida Software Developers Conference February 29, 2020 Chad Green @ChadGreen Delivering Real-Time Data with Azure 1 Chad Green Director of Software Development ScholarRx
SLIDE 2
SLIDE 3
3
@ChadGreen Delivering Real-Time Data with Azure
Agenda
Delivering Real-Time Data with Azure
- Rea
Real-Time ime Data Data and and Azur ure
- Consuming
- nsuming Data
Data Thr Throug
- ugh
h Eve Event nt Hubs ubs
- Analyz
nalyzing ing Data Data with ith Str tream m Analytics nalytics
- Ing
ngestin ting g Str treaming Data ming Data into into Power B BI
- Building
uilding Real al-Ti Time me Visualiz Visualizations ations with ith Power BI
SLIDE 4
4
@ChadGreen Delivering Real-Time Data with Azure
Real-Time Data and Azure
Delivering Real-Time Data with Azure
SLIDE 5
5
@ChadGreen Delivering Real-Time Data with Azure
Real-Time Processing
Real-Time Data and Azure
Da Data So Source ces Da Data S Stor
- rage
Ingest estion ion Stream P Process essin ing Analyt lytic ical l Da Data Sto Store Analyt lytic ics s and Report
- rtin
ing Batch Process essin ing Orchest stration ion
SLIDE 6
6
@ChadGreen Delivering Real-Time Data with Azure
- Ing
ngest, pr t, proce
- cess, and
ss, and stor store mess ssage ges in in real al-time ime
- Cannot
annot block block the the ing ingestion pipe stion pipeline line
- Act
ct on data quickly
- n data quickly
Challenges
Real-Time Data and Azure
SLIDE 7
7
@ChadGreen Delivering Real-Time Data with Azure
Architecture
Real-Time Data and Azure
Da Data So Source ces Da Data S Stor
- rage
Ingest estion ion Stream P Process essin ing Analyt lytic ical l Da Data Sto Store Analyt lytic ics s and Report
- rtin
ing Batch Process essin ing Orchest stration ion
SLIDE 8
8
@ChadGreen Delivering Real-Time Data with Azure
Architecture
Real-Time Data and Azure
Da Data So Source ces Da Data S Stor
- rage
Ingest estion ion Stream P Process essin ing Analyt lytic ical l Da Data Sto Store Analyt lytic ics s and Report
- rtin
ing Batch Process essin ing Orchest stration ion
SLIDE 9
9
@ChadGreen Delivering Real-Time Data with Azure
Architecture
Real-Time Data and Azure
Da Data So Source ces Da Data S Stor
- rage
Ingest estion ion Stream P Process essin ing Analyt lytic ical l Da Data Sto Store Analyt lytic ics s and Report
- rtin
ing Batch Process essin ing Orchest stration ion
SLIDE 10
10
@ChadGreen Delivering Real-Time Data with Azure
Architecture
Real-Time Data and Azure
Da Data So Source ces Da Data S Stor
- rage
Ingest estion ion Stream P Process essin ing Analyt lytic ical l Da Data Sto Store Analyt lytic ics s and Report
- rtin
ing Batch Process essin ing Orchest stration ion
SLIDE 11
11
@ChadGreen Delivering Real-Time Data with Azure
Architecture
Real-Time Data and Azure
Da Data So Source ces Da Data S Stor
- rage
Ingest estion ion Stream P Process essin ing Analyt lytic ical l Da Data Sto Store Analyt lytic ics s and Report
- rtin
ing Batch Process essin ing Orchest stration ion
SLIDE 12
12
@ChadGreen Delivering Real-Time Data with Azure
Technology Choices
Real-Time Data and Azure
Real-Tim ime M e Messa ssage I e Ingest estion ion Stream P Process essin ing Analyt lytic ics a s and Repor
- rtin
ing Da Data S Stor
- rage
Analytical Da Data S Stor
- re
Eve vent Hu Hubs Stream A Analyt lytic ics Power B er BI Stor
- rage B
Blob lob C Con
- ntainer
Cosm
- smos DB
DB
SLIDE 13
13
@ChadGreen Delivering Real-Time Data with Azure
Demo Scenario
Real-Time Data and Azure
- Use
ser answ answers pr practice actice te test st que question stion
- Syste
ystem m recor cords ds inte interact action ion
- Dashboar
Dashboard d sho showing ing real al-ti time me usag usage of
- f te
test st functionality unctionality
SLIDE 14
14
@ChadGreen Delivering Real-Time Data with Azure
Demo Scenario
Real-Time Data and Azure
Cl Client nt Applica lication tion
SLIDE 15
15
@ChadGreen Delivering Real-Time Data with Azure
Demo Scenario
Real-Time Data and Azure
Cl Client nt Applica lication tion
SLIDE 16
16
@ChadGreen Delivering Real-Time Data with Azure
Demo Scenario
Real-Time Data and Azure
Cl Client nt Applica lication tion
SLIDE 17
17
@ChadGreen Delivering Real-Time Data with Azure
Demo Scenario
Real-Time Data and Azure
Cl Client nt Applica lication tion
SLIDE 18
18
@ChadGreen Delivering Real-Time Data with Azure
Demo Scenario
Real-Time Data and Azure
Cl Client nt Applica lication tion
SLIDE 19
19
@ChadGreen Delivering Real-Time Data with Azure
Demo Scenario
Real-Time Data and Azure
Cl Client nt Applica lication tion
SLIDE 20
20
@ChadGreen Delivering Real-Time Data with Azure
Consuming Data Through Event Hubs
Delivering Real-Time Data with Azure
SLIDE 21
21
@ChadGreen Delivering Real-Time Data with Azure
Azure Event Hubs
Simple, secure, and scalable real-time data ingestion
Fully ully manage anaged, r d, real al-ti time me data data ing ingest stion ion se service vice that that is is sim simple ple, tr , truste usted, d, and and scalable scalable.
SLIDE 22
22
@ChadGreen Delivering Real-Time Data with Azure
Si Simp mple Sec Secure Scalab alable le Open en
Azure Event Hubs
Simple, secure, and scalable real-time data ingestion
SLIDE 23
23
@ChadGreen Delivering Real-Time Data with Azure
Event nt Produce ucer Eve vent In Inges gestion
- n
Even ent Co Consumer er Even ent Co Consumer er Even ent Co Consumer er
Event-Driven Architecture
Simple, secure, and scalable real-time data ingestion
SLIDE 24
24
@ChadGreen Delivering Real-Time Data with Azure
Throughput Units
Azure Event Hubs
Variable reserved capacities assigned to the Event Hub namespace Ingres ess 1-Mb or 1,000 events Eg Egress ss 2-Mb or 4,096 events Up to 20 TUs via Portal Up to 40 TUs via Support Ticket
SLIDE 25
25
@ChadGreen Delivering Real-Time Data with Azure
Provisioning
Consuming Data Through Event Hubs
De Demo Provision vision Azur ure Eve Event nt Hubs ubs
SLIDE 26
26
@ChadGreen Delivering Real-Time Data with Azure
Sending Data to Event Hubs
Consuming Data Through Event Hubs
De Demo Sending nding Data Data to Eve to Even H n Hubs ubs
SLIDE 27
27
@ChadGreen Delivering Real-Time Data with Azure
Analyzing Data with Stream Analytics
Delivering Real-Time Data with Azure
SLIDE 28
28
@ChadGreen Delivering Real-Time Data with Azure
Azure Stream Analytics
Serverless real-time analytics
De Delive liver po powerful ul insig insights hts from
- m
your your str streamin aming g data data with ith ease ase, in , in real al tim time.
SLIDE 29
29
@ChadGreen Delivering Real-Time Data with Azure
Potential Use Cases
Azure Stream Analytics
- Analyz
nalyze real al-tim ime te tele lemetry str streams from
- m
Io IoT device vices
- Web logs/
gs/clic ickstr tream anal am analyt ytic ics
- Ge
Geosp spatial tial anal analytic ytics s for fleet t manage manageme ment and and drive iverle less ss vehic hicle les
- Re
Remo mote mo moni nito toring and ing and predic ictiv tive mai maint ntena nance ce of hi high gh val alue as asse sets ts
- Re
Real al-time time anal analytic ytics s on n Point nt of Sal ale dat ata a for inv nventor
- ry contr
ntrol l and and ano anomaly maly detectio tion
SLIDE 30
30
@ChadGreen Delivering Real-Time Data with Azure
How does Stream Analytics work
Azure Stream Analytics
SLIDE 31
31
@ChadGreen Delivering Real-Time Data with Azure
Key Capabilities and Benefits
Azure Stream Analytics
Ea Ease se of
- f
Gettin ing g Sta Started Progr gramme mmer r Producti tivi vity ty Fully Manag anaged Low T w Total al Cos Cost of
- f
Owners rship ip (TCO) Reliabilit lity Performa rmance ce
SLIDE 32
32
@ChadGreen Delivering Real-Time Data with Azure
Key Capabilities and Benefits
Azure Stream Analytics
Ea Ease se of
- f
Gettin ing g Sta Started Progr gramme mmer r Producti tivi vity ty Fully Manag anaged Low T w Total al Cos Cost of
- f
Owners rship ip (TCO) Reliabilit lity Performa rmance ce
SLIDE 33
33
@ChadGreen Delivering Real-Time Data with Azure
Key Capabilities and Benefits
Azure Stream Analytics
Ea Ease se of
- f
Gettin ing g Sta Started Progr gramme mmer r Producti tivi vity ty Fully Manag anaged Low T w Total al Cos Cost of
- f
Owners rship ip (TCO) Reliabilit lity Performa rmance ce
SLIDE 34
34
@ChadGreen Delivering Real-Time Data with Azure
Key Capabilities and Benefits
Azure Stream Analytics
Ea Ease se of
- f
Gettin ing g Sta Started Progr gramme mmer r Producti tivi vity ty Fully Manag anaged Low T w Total al Cos Cost of
- f
Owners rship ip (TCO) Reliabilit lity Performa rmance ce
SLIDE 35
35
@ChadGreen Delivering Real-Time Data with Azure
Key Capabilities and Benefits
Azure Stream Analytics
Ea Ease se of
- f
Gettin ing g Sta Started Progr gramme mmer r Producti tivi vity ty Fully Manag anaged Low T w Total al Cos Cost of
- f
Owners rship ip (TCO) Reliabilit lity Performa rmance ce
SLIDE 36
36
@ChadGreen Delivering Real-Time Data with Azure
Key Capabilities and Benefits
Azure Stream Analytics
Ea Ease se of
- f
Gettin ing g Sta Started Progr gramme mmer r Producti tivi vity ty Fully Manag anaged Low T w Total al Cos Cost of
- f
Owners rship ip (TCO) Reliabilit lity Performa rmance ce
SLIDE 37
37
@ChadGreen Delivering Real-Time Data with Azure
Key Capabilities and Benefits
Azure Stream Analytics
Ea Ease se of
- f
Gettin ing g Sta Started Progr gramme mmer r Producti tivi vity ty Fully Manag anaged Low T w Total al Cos Cost of
- f
Owners rship ip (TCO) Reliabilit lity Performa rmance ce
SLIDE 38
38
@ChadGreen Delivering Real-Time Data with Azure
Provisioning
Analyzing Data with Stream Analytics
De Demo Creat ate a a Str tream m Analytics nalytics Job
- b
SLIDE 39
39
@ChadGreen Delivering Real-Time Data with Azure
Stream Analytics Query Language
Analyzing Data with Stream Analytics
- Aggregate
- Analytic
- Array
- GeoSpatial
- Input Metadata
- Record
- Windowing
- Scalar
SLIDE 40
40
@ChadGreen Delivering Real-Time Data with Azure
Stream Analytics Query Language
Analyzing Data with Stream Analytics
Eve Events nts and and Tim Time SELECT * FROM SensorReadings TIMESTAMP BY time SELECT System.Timestamp AS Time FROM SensorReadings
SLIDE 41
41
@ChadGreen Delivering Real-Time Data with Azure
Stream Analytics Query Language – Windowing
Analyzing Data with Stream Analytics
SELECT sensorId, COUNT(*) AS Count FROM SensorReadings TIMESTAMP BY time GROUP BY sensorId, TumblingWindow(second, 10)
SLIDE 42
42
@ChadGreen Delivering Real-Time Data with Azure
Stream Analytics Query Language – Windowing
Analyzing Data with Stream Analytics
SELECT sensorId, COUNT(*) AS Count, AVG(temp) FROM SensorReadings TIMESTAMP BY time GROUP BY sensorId, HoppingWindow(second, 10 , 5)
SLIDE 43
43
@ChadGreen Delivering Real-Time Data with Azure
Stream Analytics Query Language – Windowing
Analyzing Data with Stream Analytics
SELECT sensorId, MIN(temp) as temp FROM SensorReadings TIMESTAMP BY time GROUP BY sensorId, SlidingWindow(second, 5) HAVING MIN(temp) > 75
SLIDE 44
44
@ChadGreen Delivering Real-Time Data with Azure
Stream Analytics Query Language – Joining Multiple Steams
Analyzing Data with Stream Analytics
SELECT s1.time, s1.dspl, s1.hmdt as previousHmdt, s2.hmdt as newHmdt, datediff(ss, s1.time, s2.time) as secondsApart FROM SensorData s1 timestamp by time JOIN SensorData s2 timestamp by time ON s1.dspl = s2.dspl AND DATEDIFF(s, s1, s2) BETWEEN 0 AND 5 WHERE (s2.hmdt – s1.hmdt >= .1) or (s1.hmdt – s2.hmdt >= .1)
SLIDE 45
45
@ChadGreen Delivering Real-Time Data with Azure
Stream Analytics Query Language – Reference Data JOIN
Analyzing Data with Stream Analytics
SELECT SensorReadings.sensorID, SensorReadings.temp FROM SensorReadings JOIN thresholdRefData ON SensorReadings.sensorID = thresholdRefData.sensorID WHERE SensorReadings.temp > thresholdRefData.value
SLIDE 46
46
@ChadGreen Delivering Real-Time Data with Azure
Stream Analytics Query Language – Multiple Outputs
Analyzing Data with Stream Analytics
SELECT * INTO outputLog FROM SensorReadings SELECT * INTO outputTempAlert FROM SensorReadings WHERE temp > 75
SLIDE 47
47
@ChadGreen Delivering Real-Time Data with Azure
Querying
Analyzing Data with Stream Analytics
De Demo Que Query y a a Str tream
SLIDE 48
48
@ChadGreen Delivering Real-Time Data with Azure
Ingesting Streaming Data into Power BI
Delivering Real-Time Data with Azure
SLIDE 49
49
@ChadGreen Delivering Real-Time Data with Azure
Microsoft Power BI
Ingesting Streaming Data into Power BI
Provide vides s inte interact active ive visualiz visualizations ations and and busine business ss inte intellig lligence nce capabilitie capabilities s with ith an an inte interface ace sim simple ple enoug nough h for
- r e
end use nd users s to cr to create ate the their ir
- wn
n repor ports ts and and dashboar dashboards. ds.
SLIDE 50
50
@ChadGreen Delivering Real-Time Data with Azure
Microsoft Power BI
Ingesting Streaming Data into Power BI
Cre reate
Co Collabor borate te and and Shar hare Access Ins s Insights hts from m Anywhe nywhere
SLIDE 51
51
@ChadGreen Delivering Real-Time Data with Azure
Microsoft Power BI
Ingesting Streaming Data into Power BI
Cre reate
Co Collabor borate te and and Shar hare Access Ins s Insights hts from m Anywhe nywhere
SLIDE 52
52
@ChadGreen Delivering Real-Time Data with Azure
Microsoft Power BI
Ingesting Streaming Data into Power BI
Cre reate
Co Collabor borate te and and Shar hare Access Ins s Insights hts from m Anywhe nywhere
SLIDE 53
53
@ChadGreen Delivering Real-Time Data with Azure
Why Microsoft Power BI
Ingesting Streaming Data into Power BI
SLIDE 54
54
@ChadGreen Delivering Real-Time Data with Azure
Why Microsoft Power BI
Ingesting Streaming Data into Power BI
SLIDE 55
55
@ChadGreen Delivering Real-Time Data with Azure
Why Microsoft Power BI
Ingesting Streaming Data into Power BI
SLIDE 56
56
@ChadGreen Delivering Real-Time Data with Azure
Why Microsoft Power BI
Ingesting Streaming Data into Power BI
SLIDE 57
57
@ChadGreen Delivering Real-Time Data with Azure
Why Microsoft Power BI
Ingesting Streaming Data into Power BI
SLIDE 58
58
@ChadGreen Delivering Real-Time Data with Azure
Why Microsoft Power BI
Ingesting Streaming Data into Power BI
SLIDE 59
59
@ChadGreen Delivering Real-Time Data with Azure
Why Microsoft Power BI
Ingesting Streaming Data into Power BI
SLIDE 60
60
@ChadGreen Delivering Real-Time Data with Azure
Why Microsoft Power BI
Ingesting Streaming Data into Power BI
SLIDE 61
61
@ChadGreen Delivering Real-Time Data with Azure
Power BI Real-Time Datasets
Ingesting Streaming Data into Power BI
- Support very fast dashboard tiles
- Limited graphic options
- Uses a Redis cache under the covers
- Keeps approximately one hour of data
SLIDE 62
62
@ChadGreen Delivering Real-Time Data with Azure
Real-Time Datasets
Ingesting Streaming Data into Power BI
De Demo Creat ating ing a a Real al-Time ime Datase Dataset t with ith Str tream m Analytics nalytics
SLIDE 63
63
@ChadGreen Delivering Real-Time Data with Azure
Real-Time Datasets
Ingesting Streaming Data into Power BI
De Demo Power BI S Str treaming ming Title Titles
SLIDE 64
64
@ChadGreen Delivering Real-Time Data with Azure
Real-Time Datasets
Ingesting Streaming Data into Power BI
De Demo Rea Real-Time ime Custo ustom m Repor port t Visualiz Visualizations ations
SLIDE 65
65
@ChadGreen Delivering Real-Time Data with Azure