DRIVING INTELLIGENCE TO THE EDGE FEATURING AN END-TO-END OPEN - - PowerPoint PPT Presentation
DRIVING INTELLIGENCE TO THE EDGE FEATURING AN END-TO-END OPEN ARCHITECTURE FOR IoT Strata Data Conference, September 12, 2018 Dave Shuman, Cloudera, Industry Lead for Manufacturing and IoT Bryan Dean, Red Hat, Director Business Development
1
DRIVING INTELLIGENCE TO THE EDGE – FEATURING AN END-TO-END OPEN ARCHITECTURE FOR IoT
Strata Data Conference, September 12, 2018 Dave Shuman, Cloudera, Industry Lead for Manufacturing and IoT Bryan Dean, Red Hat, Director Business Development for IoT Solutions
2
Securely connect, authenticate and manage disparate connected devices that speak different protocols Apply analytics at the edge with machine learning and business rules to enable local, low-latency decision making Centralize IoT data processing, analytics and machine learning to enable deep business insights and actionable intelligence Create and deliver cloud-native
applications and processes. Tools to enable end-to-end data security, compliance, authorization and authentication
KEY FUNCTIONALITY FOR AN END-TO-END IoT ARCHITECTURE
Device Management & Connectivity Intelligent Edge Processing & Analytics Advanced Analytics & Machine Learning Business & Application Integration End-to-End Security & Compliance
3
WHY OPEN SOURCE FOR IoT
“We believe the best way to support this complex environment is to base
Bosch IoT Suite, on open source components and open standards. These projects establish a horizontal
the technical breeding grounds for successful business ecosystems.”
Bosch Software Innovations
million lines of code projects developers monthly visitors
4
ADDRESSING ENTERPRISE IoT NEEDS
Data Management & Analytics
Operational Technology (OT)
Information Technology (IT)
Enterprise IoT open source community
Operational Technology (OT) Information Technology (IT)
5
public, private, hybrid cloud
DATA MANAGEMENT & ANALYTICS PLATFORM IoT EDGE CONNECTED “THINGS” IoT INTEGRATION HUB
OPEN END-TO-END IoT ARCHITECTURE
APPLICATION DEVELOPMENT, DELIVERY, & INTEGRATION
Integrating IoT operating technology, data management, analytics, and applications
Cloud-native apps Traditional apps
6
CONNECTED “THINGS”
Sensors, Actuators, Data Sources Edge Processing & Analytics
DATA MANAGEMENT & ANALYTICS PLATFORM IoT INTEGRATION HUB APPLICATION DEVELOPMENT, DELIVERY, & INTEGRATION
Data Integration, Routing, Device Command/Control Application Development, Deployment, Integration Advanced Analytics & Machine Learning
EDGE PROCESSING AND ANALYTICS
IoT EDGE
decisions
7
Edge Processing & Analytics Data Integration, Routing, Device Command/Control
Telemetry Data
Application Development, Deployment, Integration Advanced Analytics & Machine Learning
DATA INTEGRATION, ROUTING, DEVICE COMMAND/CONTROL
IoT INTEGRATION HUB CONNECTED “THINGS”
Sensors, Actuators, Data Sources
DATA MANAGEMENT & ANALYTICS PLATFORM
security, access control
application platform
IoT EDGE APPLICATION DEVELOPMENT, DELIVERY, & INTEGRATION
8
Edge Processing & Analytics Data Integration, Routing, Device Command/Control Advanced Analytics & Machine Learning Application Development, Deployment, Integration
real-time analytics
IoT EDGE
ADVANCED ANALYTICS AND MACHINE LEARNING
Machine Learning Model
CONNECTED “THINGS”
Sensors, Actuators, Data Sources
DATA MANAGEMENT & ANALYTICS PLATFORM IoT INTEGRATION HUB APPLICATION DEVELOPMENT, DELIVERY, & INTEGRATION
Telemetry Data
9
Application Data Advanced Analytics & Machine Learning
APPLICATION DEVELOPMENT, DEPLOYMENT, INTEGRATION
Application Development, Deployment, Integration
Cloud-native apps Traditional apps
APPLICATION DEVELOPMENT, DELIVERY, & INTEGRATION
Edge Processing & Analytics
IoT EDGE CONNECTED “THINGS”
Sensors, Actuators, Data Sources
IoT INTEGRATION HUB
Data Integration, Routing, Device Command/Control
DATA MANAGEMENT & ANALYTICS PLATFORM
with Kubernetes orchestration
10
IoT EDGE CONNECTED “THINGS” IoT INTEGRATION HUB DATA MANAGEMENT & ANALYTICS PLATFORM
Cloud-native apps Traditional apps
OPEN END-TO-END IoT ARCHITECTURE
decisions
real-time analytics
with Kubernetes orchestration
APPLICATION DEVELOPMENT, DELIVERY, & INTEGRATION
security, access control
11
IoT EDGE CONNECTED “THINGS” IoT INTEGRATION HUB DATA MANAGEMENT & ANALYTICS PLATFORM
Cloud-native apps Traditional apps
OPEN END-TO-END IoT ARCHITECTURE: PRODUCTS
APPLICATION DEVELOPMENT, DELIVERY, & INTEGRATION
ENTERPRISE DATA HUB
12
IoT EDGE IoT INTEGRATION HUB CONNECTED “THINGS” P r
T r a n s l a t i
I n t e l l i g e n t F i l t e r i n g A g g r e g a t i
R
t i n g DATA MANAGEMENT & ANALYTICS PLATFORM
Real-Time Analytics Data Ingest Real-Time Processing Data Storage Machine Learning Data Security
Telemetry Data
Application Integration Management
Data flow to derive deep business insights and actionable intelligence
END-TO-END ANALYTICS
Telemetry Data
D e e p d a t a a n a l y s i s & i n s i g h t s
Application Data
Cloud-native apps Traditional apps
APPLICATION DEVELOPMENT, DELIVERY, & INTEGRATION
13
P r
T r a n s l a t i
I n t e l l i g e n t F i l t e r i n g A g g r e g a t i
R
t i n g D e e p d a t a a n a l y s i s & i n s i g h t s
Real-Time Analytics Data Ingest Real-Time Processing Data Storage Machine Learning Data Security
Telemetry Data
Application Integration Application Data
Data flow to derive deep business insights and actionable intelligence
END-TO-END ANALYTICS
Machine Learning
M L M
e l Actions P r e d i c t i
/ A l e r t M L M
e l IoT INTEGRATION HUB DATA MANAGEMENT & ANALYTICS PLATFORM
Cloud-native apps Traditional apps
CONNECTED “THINGS” IoT EDGE APPLICATION DEVELOPMENT, DELIVERY, & INTEGRATION
14
Red Hat Partner Confidential
PUTTING IT INTO PRODUCTION
Data Mgt. and Analytics
/home/cdsw/trained-model.h5
IoT Edge IoT Hub Connected “Things”
Predict Locally Capture Images Download Model Report Result
A trained model becomes a “downloadable brain”
15
16
END-TO-END IOT ARCHITECTURE - PoC
Application Data Telemetry Data MQTT over WiFi Telemetry Data Command & Control Telemetry Data Application Integration Client XDK
Data Mgt. and Analytics
IoT Edge
Network: Red Zone Network: Green Zone
Telemetry Data
IoT Integration Hub Applications Digital Twin
17
IoT Integration Hub
Application Data Telemetry Data MQTT over WiFi OT Middleware
OT Middleware Smart Services Machine Learning Business Logic Device Management Device Connectivity Administration Platform-as-a-Service Ingest Machine Learning Store Analyze Process
Telemetry Data Command & Control Telemetry Data Application Integration Client XDK
Digital Twin
Data Mgt. and Analytics
BI
IoT Edge Network: Red Zone Network: Green Zone
Apps CDSW PPM
Telemetry Data
END-TO-END IOT ARCHITECTURE - PoC
18 OPC- UA
Kafka Spark Kudu
Integrate Store
Spark Impala
Analyze Unified Services Layer
YARN Sentry
Data Science Workbench
TCP/IP Mqtt / OPC- UA IoT Calibrators Edge IoT Site Hub TCP/IP mqtt
decisions
security, and access control
TCP/IP Kafka
Model Deployment
DATA PIPELINE & ARCHITECTURE
real-time analytics
19
Manifold Pressure
20
Stacks for Machine Learning at the Core and Edge
CORE EDGE Machine Learning Library Machine Learning Platform Operating System Cloud / On-Prem Infrastructure Model Training Application Machine Learning Library Edge Software Library / Framework Operating System Physical Device Model Serving Application
21
So what about a Classifier?
CORE EDGE DL4J Cloudera Data Science Workbench RHEL AWS EC2 & S3 TrainClassifier.java DL4J Everyware Software Framework (Java OSGi) Yocto Linux Eurotech ReliaGATE 20-25 ServeClassifier.java Machine Learning Library Machine Learning Platform Operating System Infrastructure Model Training Application
22
Or…
CORE EDGE Keras (w/ TensorFlow backend) Cloudera Data Science Workbench RHEL Azure & ADLS train_classifier.py Tensorflow Lite (Java API) Everyware Software Framework (Java OSGi) Yocto Linux HPE Moonshot ServeClassifier.java Machine Learning Library Machine Learning Platform Operating System Infrastructure Model Training Application
23
TECHNICAL OVERVIEW: FUNCTIONALITY
IoT EDGE CONNECTED “THINGS”
Telemetry Management
IoT INTEGRATION HUB
Machine learning model
APPLICATION DEVELOPMENT, DELIVERY, & INTEGRATION
CONTAINER-BASED APPLICATION PLATFORM
Traditional applications Cloud-native applications ERP CRM DBs
DATA MANAGEMENT & ANALYTICS
Data engineering Data science Data warehouse Operational database Security & governance Machine learning Storage Services
ENTERPRISE DATA HUB
Telemetry Management App integration
CONTAINER-BASED APPLICATION PLATFORM IOT EDGE FRAMEWORK
DevOps Integration & API mgmt Orchestration Developer services
Edge analytics Machine learning
Field connectivity Cloud connectivity Edge applications Telemetry
IOT INTEGRATION FRAMEWORK
Device management Device connectivity Data integration
24
Red Hat Partner Confidential
25
TO LEARN MORE… Visit the Cloudera Booth
Featuring the End-to-end architecture and solution Join us for a Webinar on Oct 3rd to hear about the end-to-end Solution
26
27
28
Red Hat Partner Confidential
PREDOMINANT END-TO-END IoT ALTERNATIVES
Alternative Strength Weakness
Full-Stack Public Cloud Providers
Proprietary IoT Platforms
Do It Yourself
costs
29
TECHNICAL OVERVIEW: FUNCTIONALITY
IoT EDGE CONNECTED “THINGS”
Telemetry Management
IoT INTEGRATION HUB
Machine learning model
APPLICATION DEVELOPMENT, DELIVERY, & INTEGRATION
CONTAINER-BASED APPLICATION PLATFORM
Traditional applications Cloud-native applications ERP CRM DBs
DATA MANAGEMENT & ANALYTICS
Data engineering Data science Data warehouse Operational database Security & governance Machine learning Storage Services
ENTERPRISE DATA HUB
Telemetry Management App integration
CONTAINER-BASED APPLICATION PLATFORM IOT EDGE FRAMEWORK
DevOps Integration & API mgmt Orchestration Developer services
Edge analytics Machine learning
Field connectivity Cloud connectivity Edge applications Telemetry
IOT INTEGRATION FRAMEWORK
Device management Device connectivity Data integration
30
TECHNICAL OVERVIEW: PRODUCTS
IoT EDGE CONNECTED “THINGS”
Telemetry Management
IoT INTEGRATION HUB
Machine learning model
APPLICATION DEVELOPMENT, DELIVERY, & INTEGRATION
Telemetry Management App integration Telemetry
Edge analytics Machine learning
DATA MANAGEMENT & ANALYTICS
ENTERPRISE DATA HUB
31
TECHNICAL OVERVIEW: PROJECTS
IoT EDGE CONNECTED “THINGS”
Telemetry Management
IoT INTEGRATION HUB
Machine learning model
APPLICATION DEVELOPMENT, DELIVERY, & INTEGRATION DATA MANAGEMENT & ANALYTICS
Telemetry Management App integration
Edge analytics Machine learning
Telemetry
CLOUDERA’S DISTRIBUTION INCLUDING HADOOP (CDH)
32
IoT Gateways IoT Integration Hub
REMOTE MAINTENANCE & SUPPORT ON-SITE DATA ANALYTICS MACHINE LEARNING & ADVANCED ANALYTICS
Data Mgmt, Analytics & ML Manufacturing Equipment
processing
real-time analytics
real-time decisions
security, and access control
INDUSTRY 4.0 DEMO ARCHITECTURE
33
VALUE PROPOSITION
Open and interoperable
Future-proof open source architecture |
Modular
Avoid lock-in | capitalize on existing investments
End-to-end analytics
Analytics at the edge | advanced analytics and machine learning | ML model execution at the edge
Reduce risk and complexity
Simplify development, deployment, and integration tasks | save costs
End-to-end security
Security across devices, access, authentication, and applications as well as data in motion and at rest
Control your data
Privacy | security | regulatory
34
Device connectivity
Open standards – MQTT, AMQP, OPC-UA, CoAP, HTTP(s)
Flexible deployment Data management & analytics
Based on Apache open source ecosystem libraries for machine learning and advanced analytics
Open application interfaces
Enterprise visibility | real-time anomaly detection | future-proof
Community innovation
Collaboration driven by some of the leading enterprises in the IoT space Any of the leading cloud providers
No vendor lock-in
No rigid architectures or proprietary formats and components
OPEN SOURCE, OPEN STANDARDS, FLEXIBLE DEPLOYMENT