Verification Strategy & Tools for IoT Systems Design Review Team - - PowerPoint PPT Presentation

Verification Strategy & Tools for IoT Systems Design Review

Team 2013: Wissam Razouki, Long Phan, Balsha Maric Faculty Adviser: Shalabh Gupta Sponsor: Carrier Massimiliano D’Angelo and James Fan

Overview

• Introduction
• Background

○ IoT Structure ○ Project Tasks/Requirements ○ IoT Tech ○ Types of IoT Testing

• Design Approaches

○ Testing Methods ○ Framework ○ Testing Architectures ○ Types of HVAC to Consider

• Timeline
• What’s next

Introduction

• Many types of IoT (Internet of Things)

systems; need a way to test them

• Four main components of IoT system:

○ Devices - sensors push data to cloud and receive commands through gateways ○ Gateway - used to interface between sensors/actuators and cloud ○ Cloud - stores/analyzes data ○ Applications - usually mobile/web app to monitor data and/or control machine (actuators)

• Our focus: end-to-end communication between

interface and gateway, focusing on HVAC systems Carrier Wi-Fi thermostat

Simple IoT Structure

Servers used to store and analyze data, and interface with gateways/apps Telemetry - Diagnostics data from sensors Commands - Issued by apps, executed by actuators

Modified IoT Structure

Testing Architecture

Technologies Used in IoT

• RFID - uses radio frequency
• NFC - mostly used in mobile devices
• Bluetooth - close range wireless comm
• Z-Wave - low power RF tech. Mainly used for

home automation (lamp controlling, etc.)

• Wi-Fi - most common in IoT
• 2G/3G/4G/5G
• Others

RFID

Assigned Tasks from Sponsor

Task 1: Define a reference IoT system architecture Task 2: Review literature for state-of-art in IoT system verification Task 3: Synthesize proposal for IoT testing verification workflow Task 4: Toolchain selection to support the IoT testing verification workflow Task 5: (Partial) Demonstration of the proposed workflow and toolchain

Types of IoT Testing to Focus on

Our goal is to implement a system capable of performing the following:

• Interoperability testing - test how well devices interact with each other and

make use of information/data

• Stability/reliability testing - test how much the system is prone to failure,

especially in extreme conditions

• Performance testing - test speed of data transfer across entire system

(latency)

• Usability testing - test if the system is providing/can provide the required data

Challenges to Consider

• Complexity of hardware-network-software mesh
• Device interaction in real time

○ Security, backward compatibility, upgrade issues

• Real-time data testing is difficult
• Need to consider many different UIs (user interfaces)
• Operation under various network availabilities/strengths
• Many network protocols need to be tested (MQTT, CoAP, HTTP, etc)
• New field; limited research material and tools

IoT Testing Architecture - Testing-as-a-Service (TaaS)

• In this system three main testing phases are

○ Interoperability Testing ○ Conformance Testing ○ Semantics Validation Testing

• Tests are outsourced and ran sequentially
• neM2M: Machine-to-machine

standard for IoT IUT: Implementation Under Test

Cloud-based automated IoT testing

• Tests are broken up even

more

Another Example

Possible Testing Methods/Tools

• Wireshark - Network layer

○ Free and open-source software for analyzing packets sent over the internet (data integrity and transfer speeds)

• Apache JMeter - App layer

○ Java open-source software that analyze and measure the performance of web applications or a variety of services (test functional behavior)

• Build small IoT systems and run simulations - System

IoTIFY process

Types of HVAC Systems

• Heat Pump
• Rooftop Unit
• Water Source Heat Pump
• Packaged HVAC
• Split System HVAC
• Ductless System

Heat Pump

Required budget: little to none

What to do next...

Fall 2019:

• Continue research in IoT testing architectures and methods
• Meet with sponsor to discuss various architectures and pick one
• Review literature on chosen architecture

○ Modifications may be needed to satisfy our goals

• Propose a workflow

Spring 2020:

• Select the necessary tools to implement workflow

○ IoT system(s) is/are needed in order for testing to begin ○ Extensive testing and simulations required to validate our design ○ Modification to original architecture might be needed

• (Partially) Demonstrate their application on a sample IoT system

○ Final testing and video demo

References

1. https://www.guru99.com/iot-testing-challenges-tools.html 2. https://www.softwaretestinghelp.com/internet-of-things-iot-testing/ 3. https://www.buildings.com/news/industry-news/articleid/21020/title/drive-hvac-efficiency-with-internet

• of-things-functionality

4. https://www.propmodo.com/how-iot-is-taking-over-the-hvac-industry/ 5. http://twentyonecelsius.com.au/blog/everything-you-need-to-know-about-hvac-systems/ 6. https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8281514 7. https://thesai.org/Downloads/Volume10No5/Paper_45-IoT_Testing_as_a_Service.pdf 8. https://www.clariontech.com/blog/iot-testing-framework 9. https://radiostud.io/iot-testing-tools-building-resilient-iot-platforms/