AI and Big Data For Smart City in Silicon Valley, USA - Issues, - - PowerPoint PPT Presentation

A Silicon Valley Excellence Research Center Smart Technology, Computing, and Complex Systems (STCCS) – SmartTech Center San Jose State University

AI and Big Data For Smart City in Silicon Valley, USA

• Issues, Solutions, and Challenges

Presented by Jerry Gao, Ph.D., Professor, Director

SJSU and City of San Jose are teamed up as a task force for Smart Cities

A Silicon Valley Excellence Research Center Smart Technology, Computing, and Complex Systems (STCCS) – SmartTech Center San Jose State University

Building Smart City Complex Systems for San Jose

• Current Project Activities

Presented by Jerry Gao, Ph.D., Professor, Director

SJSU and City of San Jose are teamed up as a task force for Smart Cities

The Research Center Mission and Capability

“To enrich the lives of its students, to transmit knowledge to its students along with the necessary skills for applying it in the service of our society; and to expand the base of knowledge through research and scholarship.” SJSU University’s Mission:

• Provide a multi-disciplinary research platform
• Use SJSU campus and local cities as living laboratories
• Gain innovative research experience
• Learn, use, and develop cutting-edge technologies
• Solve complex issues in complex cyber systems

“To provide a multi-disciplinary research platform for SJSU faculty to create innovations and build practical and future solutions with cutting-edge technology to address the issues and challenges in building complex systems; and provide a live learning and research experience for SJSU students with rich hands-on experience and skills so that they are well-prepared to meet the future workforce needs in Silicon Valley.

Focuses:

• Research and develop sustainable technologies, intelligent solutions, and quality and safe systems

that connect objects, people, and services-based on trustworthy data using

• Validated intelligent techniques.

SJSU STCCS’s Mission:

SJSU and City of San Jose are teamed up as a task force for Smart Cities

SJSU

Multidisciplinary Research Capabilities

Smart City Complex Systems Big Data Services and Analytics Smart Sensing and Platforms IoT, Cloud and Mobile Clouds Smart Learning & Campus

Four Research Areas:

Area #3: Smart World

• Smart Resource & Recycling Systems
• Smart Green and Energy Systems
• Smart Ecological Systems
• Smart Earth Systems Engineering

Area #4: Smart Living

• Smart Home +
• Smart Food/Drink/Clothing
• Smart Healthcare
• Smart Living & Behaviors

Area #2: Smart City

• Smart Streets
• Smart Community
• Smart Transportation
• Smart Government
• Smart City as Lab
• Smart City Safety

Area #1: Smart Campus & Learning

• Smart Campus Sensor Cloud & IoT
• Smart Campus Management & Program
• Smart Interactive Learning
• Smart Campus as Lab.

Major Smart City Issues and Challenges in San Jose

Illegal Dumping How to Build Clean and Green City? Graffiti in the City How to Provide Safe and Secure City?

Where is the money?

How to build connected communities? How to construct sustainable cities? City Big Data? Where we can find?

• Controlling service for wireless camera system
• Hot-Spot station registration
• Video object collection & detection
• Data communication with the server
• Data communication with mobile APP

San Jose City Hot Spot – Illegal Dumping Mobile Station

Smart Hot-Spot Illegal Dumping Monitor System

• WiFi-based Monitor/Tracking
• Video object detection and learning
• Communication with mobile APP
• Communication with Mobile Station
• Alerting & Reporting

Mobile Services City Service Cloud

One San Jose City Street Camera-Based Trash Truck Mobile Station Mobile APP

Mobile-Edge Based Illegal Dumping Detecting & Service System

City Service Cloud

• GPS-based Monitor/Tracking
• Video object detection and learning
• Communication with mobile APP
• Communication with Mobile Station
• Alerting & Reporting
• Controlling service for wireless camera system
• Mobile station registration
• Video object collection & detection
• Data communication with the Server
• Data communication with mobile APP

Mobile Services

Smart Clean Street Assessment System Using Big Data Analytics

Level #1 Level #1 Level #2 Level #2 Level #3 Level #3

• Controlling service for wireless camera system
• Mobile station registration
• On-Land Trash Assessment on Mobile Station
• Data communication with the Server
• Data communication with mobile APP

Camera-Based Trash Truck City Service Cloud Mobile APP Mobile Station

• GPS-based Monitor/Tracking
• Video object detection and learning
• Grid-based photo object detection
• Communication with mobile APP
• Communication with Mobile Station
• Real-Time Static Assessment

Mobile Services

Smart Illegal Dumping Service System - Infrastructure

Edge-Based Trash Truck And Mobile Station San Jose City Smart City App Illegal Dump App Street Clean Monitor Car Mobile Street Sweeper truck Edge-Based Hot-Spot Mobile Station Smart Illegal Dumping Service System City Wireless Network

Major Project Objectives: Objective #1: To find out the data-driven emergency alerting coverages for six different nature disaster scenarios, such as earthquakes, floods, fire accidents, and so on. Objective #2: To find the system performance, limitations, and research ability problems in underlying emergency system infrastructures. Objective #3: To propose the ideas, enhancement approaches, and even new alerting system infrastructures and solutions for the near future. Problems: Alert System Coverage?? Alert System Performance?? Limitations of Current Systems Improvements for Future Systems

Smart City – Smart Emergency Alerting System

Major Challenges: Challenge #1: Lack of big data and lack of useful big data integration Challenge #2: Lack of built-in testing and simulation services Challenge #3: Lack of big data Legacy System with slow performance Challenge #4: Lack of effective and easy way to get big data

How to Provide Smart & Safe Living Environment?

Homes and cars are swamped

• n Last Wednesday in San Jose

Forest Fire in California California Drought Earthquake in California

Forest on Fire

camera

Real-Time Forest Fire Monitor, Analysis, and Alerting System

Satellite Based Forest FireDetection Sensor Based Forest Fire Detection

A Smart Graffiti Clean-up System Based on An Autonomous Drone

• Project Goal:
• Building a smart graffiti clean-up

system using autonomous drone using smart solution and machine learning.

• Challenges:
• Automatic graffiti detection and alerts
• Automatic graffiti clean-up
• Auto Pilot for Drone in City Street

Focused Issue: (a) Graffiti Detection and Reporting (b) Graffiti Cleaning up Major Reasons:

• High-Cost and Labor Intensive in Clean-Up
• Impact the City Image and Environment
• Affect City Traffic and Transportation

Safety

A Smart Graffiti Clean-up System Based on An Autonomous Drone

City AI and Big Data Analysis for Smart Cities

• Part I - City Illegal Dumping Object Detection

Paper and Report from: Akshay Dabholkar, Bhushan Muthiyan, Shilpa Srinivasan, Swetha Ravi, Hyeran Jeon, Jerry Gao Paper and Master Project Report from: Wei-Chung Chen, Xiaoming Chuang, Wendy Hu, Luwen Miao, and Jerry Gao

• Part II - Street Litter Object Detection and Framework

Paper and Master Project Report from: Chandni Balchandani, Rakshith Koravadi Hatwar, Parteek Makkar, Yanki Shah, Pooja Yelure, Magdalini Eirinaki Paper and Master Project Report from: Bharat Bhushan Kavin Pradeep Sriram Kumar Mithra Desinguraj Sonal Gupta, and Jerry Gao

Edge-Based Mobile Smart Service System for Illegal Dumping

Edge-Based Trash Truck And Mobile Station San Jose City Smart City App Illegal Dump App Street Clean Monitor Car Mobile Street Sweeper truck Edge-Based Hot-Spot Mobile Station Smart Illegal Dumping Service System City Wireless Network Hot Spot

Illegal dumping

Mobile Station

Illegal Dumping App

Smart Mobile Illegal Dumping Service System (SMIDS)

Illegal Dumping Service Server

Illegal Dumping Service Manager

Edge-Based Mobile Station

Illegal Dumping Service DB Program Illegal Dumping Detection Engine Illegal Dumping Service Connector Illegal Dumping Mobile Client IoT Mobile Platform & Sensors Computing Vision & Object Detection Illegal Dumping Controller Illegal Dumping Reporter Illegal Dumping Analytics Illegal Dumping Service Protocol Illegal Dumping Server UI Illegal Dumping Service Protocol Edge-Based Data Repository Illegal Dumping Dashboard Mobile Edge Computing Platform QoS & Security QoS & Security Crime/Safty

Edge-Based Automatic City Illegal Dumping Object Detection

Group #1

AI Model and Technology

• Convolutional Neural Network (CNN) is

a practically useful algorithm especially in computer vision tasks by reduced complexity with resolving

• verfitting problems from deep

learning.

• CNN inside is made of simple repeated

matrix multiplications without branch

• perations
• Create illegally dumped material labeled dataset.
• Create lmdb database from the dataset.
• Tweak the existing model as per need and train the model using
• Caffe interface with the generated lmdb files.
• Deploy the model file on the embedded platform Jetson TX1
• Test the model for prediction accuracy.

Illegal Dumping Object Detection Approaches

Approach 1 (Naive Approach) Frequently dumped wastes identified in 102 illegal dumping images provided by City of San Jose

• 6 classes
• Baseline Network: AlexNet
• Iterations: 100
• Image dataset size:161 Mb

0% 10% 20% 30% 40% 50% 60% 70%

Chair Mattress Table Furniture Sofa Trash

Prediction Accuracy of Approach 1

Illegal Dumping Object Detection Approaches

Approach 2 Adding more classes based on Approach #1:

• 11 classes
• Baseline Network: AlexNet and Googlenet
• Iterations: 5000
• Image dataset size: 802 Mb

0% 20% 40% 60% 80% 100% 120%

Cart Chair Clean Area Electronics Furniture Mattress Sofa Table Trash Trash Bags Tree GoogLeNet AlexNet

The disadvantage of this approach was the model tends to predict clean area even for the image with illegal dump object because the feature map for clean areas was more compared to normal images. Prediction Accuracy of Approach 2

Illegal Dumping Object Detection Approaches

Approach 3: Using pre-processed images

• Cropping image dataset for better training.
• 8 classes
• Baseline Network: AlexNet and Googlenet
• Iterations: 10000
• Image dataset size: 1.14 Gb

Solution improvement:

• Image pre-processing to clearly

define the region of interest.

• Excluding Clean Area

classification.

• Classes Aggregation

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Cart Electronics Furniture Mattress Sofa Trash Trash Bags Tree GoogLeNet(%) AlexNet(%)

Prediction Accuracy of Approach 3

Illegal Dumping Object Detection Approaches

Approach 4: Energy Efficiency Approach- TensorRT

• We used NVIDIA TensorRT that helps shrink model size.
• Uses two approaches to shrink trained model size.
• 1. Quantization,
• 2. Optimizations by applying vertical and horizontal layer fusion.

Illegal Dumping Object Detection Approaches

227.6 227.7 41.4 227.6 41.3 84.7 113.8 12 113.8 12 50 100 150 200 250 300 AlexNet AlexNet GoogleNet AlexNet GoogleNet Approach 1 Approach 2 Approach 3 Original (in MB) With TensorRT (in MB) Approaches Network Original (in MB) With TensorRT (in MB) Approach 1 AlexNet 227.6 84.7 Approach 2 AlexNet 227.7 113.8 GoogleNet 41.4 12 Approach 3 AlexNet 227.6 113.8 GoogleNet 41.3 12

Illegal Dumping Detecting Engine (By Group #2)

Illegal Dumping Object Detection Engine – Case Study I

Conclusions:

• Great to have localization but at the cost of classification accuracy
• Localization before classification, localization capability determines the classification ability

Experimental design:

• Train Faster R-CNN using VGG from scratch in Caffe;
• Localization (represented by bounding boxes) and classifications (represented by a

class label and probability)

Illegal Dumping Object Detection Engine – Case Study II

Conclusions:

• Decent three-class classification accuracy
• Does not handle different viewpoints well possibly because the training set does not

contain enough images from all viewpoints.

• Does not handle multiple objects well possibly because the training set does not

contain enough images with multiple objects Experimental design:

• Retrain the last layer of Inception v3 as a softmax in TensorFlow
• Number of classes = 3
• Images with slight duplicate images of different sizes (data augmentation)

Illegal Dumping Object Detection Engine – Case Study III

Experimental design:

• a. Retrain the last layer of Inception v3 as a softmax in TensorFlow
• b. Number of classes = 3

Conclusions:

• Overfitting is an issue because the final test accuracy is 100%
• Possible solutions to prevent overfitting are: collect more images and apply

regularization (L2 or dropout)

Illegal Dumping Object Detection Engine – Case Study V

Conclusion for case study 5:

• Better seven-class classification accuracy
• To improve the accuracy, we could use more images and increase the intra-

class variations of each category Experimental design:

• Retrain the last layer of Inception v3 as a softmax in TensorFlow
• Number of classes = 7

Illegal Dumping Object Detection Engine – Case Study IV

Conclusion for case study 4:

• Good seven-class classification accuracy
• To improve the accuracy, we could use more images and increase the intra-class

variations of each category Experimental design:

• Retrain the last layer of Inception v3 as a softmax in TensorFlow
• Number of classes = 7
• Images with slight duplicate images of different sizes (data augmentation)

Illegal Dumping Object Detection Engine – Case Study V

Conclusion for case study 5:

• Better seven-class classification accuracy
• To improve the accuracy, we could use more images and increase the intra-

class variations of each category Experimental design:

• Retrain the last layer of Inception v3 as a softmax in TensorFlow
• Number of classes = 7

Illegal Dumping Object Detection Engine – Case Study V

A Street Cleanliness Assessment System for Smart City using Machine Learning and Cloud

Objective:

• Support real-time automatic digital

data-driven analysis, tracking, and monitoring of city street cleanliness

• Provide automatic response,

management, service solutions for city cleanliness

• Offer a real-time dashboard of city

Mayor and public about city street environment and cleanliness based on digital cleanliness protocols and standards.

Digital Colored City Street Map Digital Colored City Street Digital Colored City Street Block

Street Cleaning Monitoring

System Architecture

Street Cleaning UI Street Cleaning Dashboard Street Cleaning Reports Streets Blocks Mobile Stations Street Cleaning Detection Engine Street Cleaning Detection Analytics Street Cleaning DB service

Engine DB (NoSQL) Application DB (MySQL)

Street Cleaning Service Protocols

Mobile Station Connection Module ServiceRequest Module

Street Cleaning Service Manager Admin Feedback Dispatch

DB Connection Control Module UI Connection Module

Mobile Client (MS) Controller MS Computing MS Monitoring MS Repo Street Cleaning Security MS Security ACL/Authentication Encryption/Session Mgmt. Role Based Authorization. Performance Alerts Historical

Highlights

• Distributed hybrid deep learning based

image processing pipeline.

• Deep Learning Framework/Model Agnostic

Phases.

• On Demand Scaling based on the volume of

input images.

• Easily extend object clusters/classes to

detect new objects.

• Embed localization and classification

information inside image metadata (exif).

• Real Time dashboard visualizing the

cleanliness of the streets.

• Reduce the operational cost and Optimize

resource allocation.

Challenges

• Blurred Images.
• Perspective of Photos.
• Making the pipeline to respond in real

time.

• Optimizing the models to run in

resource constrained environment for edge processing.

Dependencies

• High specification Server with GPU

support, powerful processor and large memory capacity.

• Availability of Training Data.

Contribution

• Deep Learning-based framework for litter

detection and classification.

• Capture street images using garbage truck

mounted cameras.

• Send to image processing pipeline: multiple

phases where litter objects are detected and segmented.

• Group images by geo-location to give a

coherent view of the cleanliness of the street.

• Display results on interactive dashboard.

Pipeline

• Curb and Street Detection and

Localization.

• Obstacle Detection (Cars, People)
• Object Detection and Clustering

into high level classes(Glass, Metal, Liquid).

• Object Classification (Bottles,

Cans, Leaves)

• Result Integration

Dashboard View

City Street View

NewsFeeds View

Clean-up Service Requests View

Manage Cleanup Crew View

Notifications View

System Implementation

Original Image

• Input by user

Phase 1

• Street Detection

(Deepmask) Phase 2

• Object Detection

(Deepmask) Phase 3

• Object Classification

(Tensor Flow)

Phase Outputs

Image Annotation View

Annotation Tool – Edit View Dashboard

Image Pipeline View

Dashboard Detailed View

50 100 150 200 250 1 3 5

Phase Performance (In Seconds)

Phase 1 Phase 2 Phase 3 0% 50% 100% 150% Cleanliness Prediction (Clean Street Images) Expected Actual Average 0% 50% 100% Dirty_1 Dirty_2 Dirty_3 Dirty_4 Dirty_5 Dirty_6 Dirty_7 Dirty_8

Cleanliness Prediction (Images with Little Litter)

Expected Actual 0% 20% 40% 60% 80% 100%

Cleanliness Prediction (Images with Little Litter)

Expected Actual Average

Phase wise performance of pipeline for images with varying level of cleanliness. Red bar – Phase 1, Orange – Phase 2 and Green – Phase 3. Cleanliness score computed for 10 clean streets. Correct classified instances : 8 Accuracy = 8/10 = 80% Cleanliness score computed for 8 streets with little litter(dirty streets). Correct classified instances : 5 Accuracy = 5/8 = 62.5% Cleanliness score computed for 10 streets with a lot of litter (very dirty streets). Correct classified instances : 8 Accuracy = 8/10 = 80%

Future Research Directions, Challenges, and Needs for Smart Cities

A Silicon Valley Excellence Research Center Smart Technology, Computing, and Complex Systems (STCCS) – SmartTech Center San Jose State University

Presented by Jerry Gao, Ph.D., Professor, Director

SJSU and City of San Jose are teamed up as a task force for Smart Cities

How to Build Connected Smart Cities?

Social networks IT networks Sensor Networks/Clouds Wireless networks and mobile clouds City Government Smart Home Community & Neighborhood

Intellectual Merit:

• Tasks 1: Multi-modal Communication

and Network Interoperability – Multi-modal, reliability, interoperability

• Tasks 2: Software-defined Edge Cloud

– Remote control, orchestration, SDN policy engine

• Tasks 3: Mobile Enabled Resilient

Quality Test and Automatic Validation – Seamless emergency testing service, large-scale automatic repeatable test, QoS Broader Impacts

• Task1: Cloud dashboard

Help multiple departments to better manage the rescue and relief resources

• Task2: Communities

Provide a robust connectivity solution, enable rapid and effective emergency response

Switch Router Switch Basestation

Internet Backbone

Gateway Gateway

X

Link break

X

Link break Community Gateway WiFi Moving Gateway WiFi

Legend

Link Break Internet Peer to Peer Link

X X

User Gateway

X

Internet Service is down Community Network

Switch

WiFi Internet Service is normal

X

NSF EAGER Grant - Creating a Community Infrastructure for Interoperable Emergency Connectivity (for City Community)

Smart and Connected City Neighborhoods and Communities

City Hall and Departments City Library Community Center City Neighborhoods Programs/Organizations/Services) School Hospitals Businesses People Different Groups

Smart City Big Data

Transportation & Traffic Big Data Environment Big Data City Cyber Information Infrastructure

Smart City Big Data

City Emergency preparedness Big Data Community Big Data Networking & Mobility Big Data City IoT Big Data City Government Big Data

How to Build Sustainable Smart Cities?

Sustainable Transportation Green & Sustainable Living Resource Sustainable Infrastructure Sustainable Cyber Infrastructure Sustainable Economic/Business

Smart City Big Data Challenges

It has a big data junk yard No well-defined service-oriented big data platform for smart cities Big Data Ownership??

Smart City Big Data

Lack of well-defined Big Data Warehouses with Quality Assurance Solutions Big Data Quality and Certification

City AI and Big Data Research Needs and Challenges

AI Dynamic Modeling Transportation Traffic Behavior Modeling Mobile Object Modeling Dynamic Environment Modeling City Safety Modeling People Community Behavior Modeling People Dynamic Behavior Modeling

Challenges and Needs in Building Smart City Complex Systems Smart City Big Data

Issue #1: Too many isolated information islands  flexible, systematic information classification and integration Issue #2: How to build sustainable smart city system infrastructures?  Systematic, sustainable, software-defined solution for IoT infrastructures, network infrastructures, cloud and mobile cloud infrastructures. Issue #3: Where are the service-oriented open platforms/framework for smart city system development  FiWare, …..? Issue #4: Where are the smart city big data service platform to support diverse smart city systems and solutions?  No found? Issue #5: Trustworthy smart city big data  Smart city open big data? public open data?  New big data businesses and production solutions for trustworthy smart city big data  Well-defined quality assurance standards and quality validation service platforms

SJSU

IEEE Joint International Conferences on Smart World and Smart City in Silicon Valley

• IEEE Smart World Congress 2017 in Silicon Valley

URL: http://ieee-smartworld.org/2017/smartworld/ IEEE Smart World Congress 2017 IEEE Smart City Innovation 2017

• IEEE Smart City Innovation 2017  August, 2017
• Conjunction with IEEE SmartWorld 2017, San Francisco Bay, USA

URL: http://ieee-smartworld.org/2017/sci/

SJSU

Related References

Autonomous UAV Forced Graffiti Detection and Removal System Based on Machine Learning

• Aug 2017
• 2017 IEEE International Conference on Smart City and Innovation

A Practical Study on Quality Evaluation for Age Recognition Systems

• Aug 2017
• SEKE2017 - The Twenty-Ninth International Conference on Software Engineering and Knowledge

Engineering An Edge-Based Smart Mobile Service System for Illegal Dumping Detection and Monitoring in San Jose

• Aug 2017
• 2017 IEEE International Conference on Smart City and Innovation

Data-driven Forest Fire analysis

• Aug 2017
• 2017 IEEE International Conference on Smart City and Innovation

A Survey on Quality Assurance Techniques for Big Data Applications

• Apr 2017
• IEEE BigDataService 2017 - International Workshop on QUALITY ASSURANCE AND

VALIDATION On Building a Big Data Analysis System for California Drought

• Apr 2017
• IEEE BigDataService 2017, San Francisco, April 7-10, 2017

Related References

Big Data Validation Case Study Apr 2017 IEEE BigDataService 2017 - International Workshop on QUALITY ASSURANCE AND VALIDATION Data-Driven Water Quality Analysis and Prediction: A Survey Apr 2017 The 3rd IEEE International Conference on BIG DATA COMPUTING SERVICE AND APPLICATIONS Bike-Sharing System: A Big-Data Perspective Jan 2017, Smart Computing and Communication Quality Assurance for Big Data Applications– Issues, Challenges, and Needs Jul 2016 The Twenty-Eighth International Conference on Software Engineering and Knowledge Engineering On Building a Big Data Analysis System for California Drought

• Apr 2017
• IEEE BigDataService 2017, San Francisco, April 7-10, 2017

Big Data Validation and Quality Assurance – Issuses, Challenges, and Needs

• Mar 2016
• IEEE 10th IEEE International Symposium on Service-Oriented System Engineering

Data-Driven Water Quality Analysis and Prediction: A Survey

• Apr 2017
• The 3rd IEEE International Conference on BIG DATA COMPUTING SERVICE AND

APPLICATIONS