NDN, CoAP, and MQTT: A Comparative Measurement Study in the IoT
ACM ICN 2018, Boston Cenk Gündoğan1 Peter Kietzmann1 Martine Lenders2 Hauke Petersen2 Thomas C. Schmidt1 Matthias Wählisch2
1HAW Hamburg 2Freie Universität Berlin
NDN, CoAP, and MQTT: A Comparative Measurement Study in the IoT ACM - - PowerPoint PPT Presentation
NDN, CoAP, and MQTT: A Comparative Measurement Study in the IoT ACM ICN 2018, Boston Cenk Gndoan 1 Peter Kietzmann 1 Martine Lenders 2 Hauke Petersen 2 Thomas C. Schmidt 1 Matthias Whlisch 2 1 HAW Hamburg 2 Freie Universitt Berlin CoAP
ACM ICN 2018, Boston Cenk Gündoğan1 Peter Kietzmann1 Martine Lenders2 Hauke Petersen2 Thomas C. Schmidt1 Matthias Wählisch2
1HAW Hamburg 2Freie Universität Berlin
▶ Sensors and actuators connect to clouds
▶ Today mainly based on MQTT
MQTT
Publish–subscribe protocol on TCP Constrained IoT: MQTT-SN on UDP
CoAP
Request–response protocol on UDP IETF solution for constrained IoT
ICN
Promising candidate for future IoT Intensifying attention since 2014
Cloud
report control Sensor nodes
2 / 56
▶ Sensors and actuators connect to clouds
▶ Today mainly based on MQTT
▶ MQTT
▶ Publish–subscribe protocol on TCP ▶ Constrained IoT: MQTT-SN on UDP
CoAP
Request–response protocol on UDP IETF solution for constrained IoT
ICN
Promising candidate for future IoT Intensifying attention since 2014
Cloud
report control Sensor nodes
3 / 56
▶ Sensors and actuators connect to clouds
▶ Today mainly based on MQTT
▶ MQTT
▶ Publish–subscribe protocol on TCP ▶ Constrained IoT: MQTT-SN on UDP
▶ CoAP
▶ Request–response protocol on UDP ▶ IETF solution for constrained IoT
ICN
Promising candidate for future IoT Intensifying attention since 2014
Cloud
report control Sensor nodes
4 / 56
▶ Sensors and actuators connect to clouds
▶ Today mainly based on MQTT
▶ MQTT
▶ Publish–subscribe protocol on TCP ▶ Constrained IoT: MQTT-SN on UDP
▶ CoAP
▶ Request–response protocol on UDP ▶ IETF solution for constrained IoT
▶ ICN
▶ Promising candidate for future IoT ▶ Intensifying attention since 2014
Cloud
report control Sensor nodes
5 / 56
6 / 56
Protocol Overview Metrics & Experiment Setup Evaluation Single-hop: Push vs. Pull Protocols Multi-hop: Reliability and Protocol Performance Conclusion & Outlook
7 / 56
GET /temp 2.05 Content 21◦C
GET /temp GET /temp 2.05 Content 21 C PUT /temp 21 C 2.04 Changed
PUT /temp 21 C PUT /temp 21 C 2.04 Changed GET /temp [OBS] 2.05 Content 21 C . . . 2.05 Content 22 C
8 / 56
GET /temp 2.05 Content 21◦C
X GET /temp GET /temp 2.05 Content 21 C PUT /temp 21 C 2.04 Changed
PUT /temp 21 C PUT /temp 21 C 2.04 Changed GET /temp [OBS] 2.05 Content 21 C . . . 2.05 Content 22 C
9 / 56
GET /temp 2.05 Content 21◦C
X GET /temp 2.05 Content 21◦C GET /temp 2.05 Content 21 C PUT /temp 21 C 2.04 Changed
PUT /temp 21 C PUT /temp 21 C 2.04 Changed GET /temp [OBS] 2.05 Content 21 C . . . 2.05 Content 22 C
10 / 56
GET /temp 2.05 Content 21◦C
X GET /temp 2.05 Content 21◦C GET /temp 2.05 Content 21◦C PUT /temp 21 C 2.04 Changed
PUT /temp 21 C PUT /temp 21 C 2.04 Changed GET /temp [OBS] 2.05 Content 21 C . . . 2.05 Content 22 C
11 / 56
GET /temp 2.05 Content 21◦C
X GET /temp 2.05 Content 21◦C GET /temp 2.05 Content 21◦C PUT /temp 21◦C 2.04 Changed
PUT /temp 21 C PUT /temp 21 C 2.04 Changed GET /temp [OBS] 2.05 Content 21 C . . . 2.05 Content 22 C
12 / 56
GET /temp 2.05 Content 21◦C
X GET /temp 2.05 Content 21◦C GET /temp 2.05 Content 21◦C PUT /temp 21◦C 2.04 Changed
X PUT /temp 21◦C PUT /temp 21 C 2.04 Changed GET /temp [OBS] 2.05 Content 21 C . . . 2.05 Content 22 C
13 / 56
GET /temp 2.05 Content 21◦C
X GET /temp 2.05 Content 21◦C GET /temp 2.05 Content 21◦C PUT /temp 21◦C 2.04 Changed
X PUT /temp 21◦C 2.04 Changed PUT /temp 21 C 2.04 Changed GET /temp [OBS] 2.05 Content 21 C . . . 2.05 Content 22 C
14 / 56
GET /temp 2.05 Content 21◦C
X GET /temp 2.05 Content 21◦C GET /temp 2.05 Content 21◦C PUT /temp 21◦C 2.04 Changed
X PUT /temp 21◦C 2.04 Changed PUT /temp 21◦C 2.04 Changed GET /temp [OBS] 2.05 Content 21 C . . . 2.05 Content 22 C
15 / 56
GET /temp 2.05 Content 21◦C
X GET /temp 2.05 Content 21◦C GET /temp 2.05 Content 21◦C PUT /temp 21◦C 2.04 Changed
X PUT /temp 21◦C 2.04 Changed PUT /temp 21◦C 2.04 Changed GET /temp [OBS] 2.05 Content 21◦C . . . 2.05 Content 22◦C
16 / 56
CONNECT CONNACK REGISTER /temp REGACK topicid SUBSCRIBE /temp SUBACK
PUBLISH topicid 21 C
PUBLISH topicid 21 C PUBLISH topicid 21 C PUBACK
17 / 56
CONNECT CONNACK REGISTER /temp REGACK topicid SUBSCRIBE /temp SUBACK
PUBLISH topicid 21◦C
PUBLISH topicid 21 C PUBLISH topicid 21 C PUBACK
18 / 56
CONNECT CONNACK REGISTER /temp REGACK topicid SUBSCRIBE /temp SUBACK
PUBLISH topicid 21◦C
X PUBLISH topicid 21◦C PUBLISH topicid 21 C PUBACK
19 / 56
CONNECT CONNACK REGISTER /temp REGACK topicid SUBSCRIBE /temp SUBACK
PUBLISH topicid 21◦C
X PUBACK PUBLISH topicid 21◦C PUBLISH topicid 21 C PUBACK
20 / 56
CONNECT CONNACK REGISTER /temp REGACK topicid SUBSCRIBE /temp SUBACK
PUBLISH topicid 21◦C
X PUBACK PUBLISH topicid 21◦C PUBLISH topicid 21◦C PUBACK
21 / 56
Interest /temp Data /temp 21◦C Interest /temp Interest /temp Data /temp 21 C
ACK ACK ACK
Interest /temp/21 C Data /temp/21 C [ACK] Interest /temp/21 C
ACK ACK ACK
Interest /temp/21 C Data /temp/21 C [ACK] NAM Interest /temp Data /temp 21 C NAM Interest /temp Data /temp 21 C
22 / 56
Interest /temp Data /temp 21◦C X Interest /temp Interest /temp Data /temp 21 C
ACK ACK ACK
Interest /temp/21 C Data /temp/21 C [ACK] Interest /temp/21 C
ACK ACK ACK
Interest /temp/21 C Data /temp/21 C [ACK] NAM Interest /temp Data /temp 21 C NAM Interest /temp Data /temp 21 C
23 / 56
Interest /temp Data /temp 21◦C X Interest /temp Data /temp 21◦C Interest /temp Data /temp 21 C
ACK ACK ACK
Interest /temp/21 C Data /temp/21 C [ACK] Interest /temp/21 C
ACK ACK ACK
Interest /temp/21 C Data /temp/21 C [ACK] NAM Interest /temp Data /temp 21 C NAM Interest /temp Data /temp 21 C
24 / 56
Interest /temp Data /temp 21◦C X Interest /temp Data /temp 21◦C Interest /temp Data /temp 21◦C
ACK ACK ACK
Interest /temp/21 C Data /temp/21 C [ACK] Interest /temp/21 C
ACK ACK ACK
Interest /temp/21 C Data /temp/21 C [ACK] NAM Interest /temp Data /temp 21 C NAM Interest /temp Data /temp 21 C
25 / 56
Interest /temp Data /temp 21◦C X Interest /temp Data /temp 21◦C Interest /temp Data /temp 21◦C
ACK ACK ACK
Interest /temp/21◦C Data /temp/21◦C [ACK] Interest /temp/21 C
ACK ACK ACK
Interest /temp/21 C Data /temp/21 C [ACK] NAM Interest /temp Data /temp 21 C NAM Interest /temp Data /temp 21 C
26 / 56
Interest /temp Data /temp 21◦C X Interest /temp Data /temp 21◦C Interest /temp Data /temp 21◦C
ACK ACK ACK
Interest /temp/21◦C Data /temp/21◦C [ACK] X Interest /temp/21◦C
ACK ACK ACK
Interest /temp/21 C Data /temp/21 C [ACK] NAM Interest /temp Data /temp 21 C NAM Interest /temp Data /temp 21 C
27 / 56
Interest /temp Data /temp 21◦C X Interest /temp Data /temp 21◦C Interest /temp Data /temp 21◦C
ACK ACK ACK
Interest /temp/21◦C Data /temp/21◦C [ACK]
ACK ACK
X Interest /temp/21◦C Data /temp/21◦C [ACK]
ACK ACK ACK
Interest /temp/21 C Data /temp/21 C [ACK] NAM Interest /temp Data /temp 21 C NAM Interest /temp Data /temp 21 C
28 / 56
Interest /temp Data /temp 21◦C X Interest /temp Data /temp 21◦C Interest /temp Data /temp 21◦C
ACK ACK ACK
Interest /temp/21◦C Data /temp/21◦C [ACK]
ACK ACK
X Interest /temp/21◦C Data /temp/21◦C [ACK]
ACK ACK ACK
Interest /temp/21◦C Data /temp/21◦C [ACK] NAM Interest /temp Data /temp 21 C NAM Interest /temp Data /temp 21 C
29 / 56
Interest /temp Data /temp 21◦C X Interest /temp Data /temp 21◦C Interest /temp Data /temp 21◦C
ACK ACK ACK
Interest /temp/21◦C Data /temp/21◦C [ACK]
ACK ACK
X Interest /temp/21◦C Data /temp/21◦C [ACK]
ACK ACK ACK
Interest /temp/21◦C Data /temp/21◦C [ACK] NAM Interest /temp Data /temp 21 C NAM Interest /temp Data /temp 21 C
30 / 56
Interest /temp Data /temp 21◦C X Interest /temp Data /temp 21◦C Interest /temp Data /temp 21◦C
ACK ACK ACK
Interest /temp/21◦C Data /temp/21◦C [ACK]
ACK ACK
X Interest /temp/21◦C Data /temp/21◦C [ACK]
ACK ACK ACK
Interest /temp/21◦C Data /temp/21◦C [ACK] NAM Interest /temp Data /temp 21◦C NAM Interest /temp Data /temp 21 C
31 / 56
Interest /temp Data /temp 21◦C X Interest /temp Data /temp 21◦C Interest /temp Data /temp 21◦C
ACK ACK ACK
Interest /temp/21◦C Data /temp/21◦C [ACK]
ACK ACK
X Interest /temp/21◦C Data /temp/21◦C [ACK]
ACK ACK ACK
Interest /temp/21◦C Data /temp/21◦C [ACK] NAM Interest /temp Data /temp 21◦C NAM Interest /temp Data /temp 21◦C
32 / 56
Current IoT Protocols ICN Protocols CoAP MQTT-SN NDN I-Not HoPP GET PUT OBS Transport UDP UDP UDP UDP — — — Push ✘ ✓ ✓ ✓ ✘ ✓ ✘ Pull ✓ ✘ ✘ ✘ ✓ ✘ ✓ Pub/Sub ✘ ✘ ✓ ✓ ✘ ✘ ✓ Flow Control ✘ ✘ ✘ ✘ ✓ ✘ ✓ Reliability (c) (c) ✘ (Q1) ✓ ✓ ✓
33 / 56
34 / 56
▶ CoAP: gcoap ▶ MQTT-SN: asymcute ▶ On top of UDP/IPv6 ▶ NDN v0.2 ▶ HoPP & I-Not (extensions) ▶ On top of link layer
35 / 56
▶ Network stack sizes ▶ Time to content arrival ▶ Link stress ▶ Goodput ▶ Network utilization (control vs data plane) ▶ Energy consumption ▶ Security overhead
36 / 56
▶ Network stack sizes ▶ Time to content arrival ▶ Link stress ▶ Goodput ▶ Network utilization (control vs data plane) ▶ Energy consumption ▶ Security overhead
37 / 56
IoT-Lab Testbed ARM Cortex-M3 64 kB RAM & 512 kB ROM IEEE 802.15.4 radio CSMA/CA Retransmissions: Link layer: 4 × 2–10 ms Network layer: 4 × 2 s Single-hop ▶ 1 gateway + 1 IoT node Multi-hop ▶ 1 gateway + 50 IoT nodes ▶ Max. hop distance: 6
38 / 56
39 / 56
N D N H
P I
M Q T T
N C
P G E T C
P P U T C
P O B S 10 20 30 40 50 [KiB] N D N H
P I
M Q T T
N C
P G E T C
P P U T C
P O B S 10 20 30 40 50 [KiB] App Heap CCN-lite IPv6 OS
40 / 56
Time to content arrival for scheduled publishing every 50 ms
5 10 15 20 Time to Completion [ms] 0.2 0.4 0.6 0.8 1.0 CDF
5 10 15 20 Time to Completion [ms] 0.2 0.4 0.6 0.8 1.0 CDF
41 / 56
Time to content arrival for scheduled publishing every 50 ms
5 10 15 20 Time to Completion [ms] 0.2 0.4 0.6 0.8 1.0 CDF
MQTT-SN (Q0) MQTT-SN (Q1)
5 10 15 20 Time to Completion [ms] 0.2 0.4 0.6 0.8 1.0 CDF
42 / 56
Time to content arrival for scheduled publishing every 50 ms
5 10 15 20 Time to Completion [ms] 0.2 0.4 0.6 0.8 1.0 CDF
CoAP OBS CoAP PUT (n) CoAP PUT (c) MQTT-SN (Q0) MQTT-SN (Q1)
5 10 15 20 Time to Completion [ms] 0.2 0.4 0.6 0.8 1.0 CDF
CoAP GET (n) CoAP GET (c)
43 / 56
Time to content arrival for scheduled publishing every 50 ms
5 10 15 20 Time to Completion [ms] 0.2 0.4 0.6 0.8 1.0 CDF
I-Not CoAP OBS CoAP PUT (n) CoAP PUT (c) MQTT-SN (Q0) MQTT-SN (Q1)
5 10 15 20 Time to Completion [ms] 0.2 0.4 0.6 0.8 1.0 CDF
CoAP GET (n) CoAP GET (c) NDN HoPP
44 / 56
Takeaways
▶ Without network layer repair
▶ Push protocols complete below 10 ms ▶ Pull protocols double completion time
▶ With network layer repair
▶ Reliability layer increases completion time to seconds ▶ Retransmissions may induce additional link stress
45 / 56
Time to content arrival for unscheduled publishing every [1 … 3] s, content request every 1 s
5 10 15 20 Time to Completion [ms] 0.2 0.4 0.6 0.8 1.0 CDF
2 4 6 8 10 Time to Completion [s] 0.2 0.4 0.6 0.8 1.0 CDF
46 / 56
Time to content arrival for unscheduled publishing every [1 … 3] s, content request every 1 s
5 10 15 20 Time to Completion [ms] 0.2 0.4 0.6 0.8 1.0 CDF
I-Not CoAP OBS CoAP PUT (n) CoAP PUT (c) MQTT-SN (Q0) MQTT-SN (Q1)
2 4 6 8 10 Time to Completion [s] 0.2 0.4 0.6 0.8 1.0 CDF CoAP GET (n) CoAP GET (c) NDN HoPP
12 14 16 18 20 [ms]
47 / 56
▶ Interests arrive at higher rate than content
⇒ Open PIT states accumulate
▶ PIT size very limited in constrained IoT scenario ▶ Two possible strategies:
▶ Discard newly arriving Interests ▶ Overwrite PIT states ⇒ Leads to delays or loss
48 / 56
▶ Radio interference ▶ Additional link- & network-layer retransmissions ▶ Higher link stress
49 / 56
Single-hop (50 ms) Multi-hop (5 s)
Success [%]
Success [%] Unreliable IP 8.0 97 0.03 57 Reliable IP 305.0 99 3.83 78 I-Not 7.2 100 1.98 68 NDN & HoPP 13.6 100 0.60 98
Protocol
Protocol Unreliable IP 30.6 CoAP PUT (n) 0.2 CoAP GET (n) Reliable IP 5000.0 CoAP PUT (c) 16.3 CoAP PUT (c) NDN & HoPP 27.9 HoPP 13.7 HoPP
50 / 56
Single-hop (50 ms) Multi-hop (5 s)
Success [%]
Success [%] Unreliable IP 8.0 97 0.03 57 Reliable IP 305.0 99 3.83 78 I-Not 7.2 100 1.98 68 NDN & HoPP 13.6 100 0.60 98
Protocol
Protocol Unreliable IP 30.6 CoAP PUT (n) 0.2 CoAP GET (n) Reliable IP 5000.0 CoAP PUT (c) 16.3 CoAP PUT (c) NDN & HoPP 27.9 HoPP 13.7 HoPP
51 / 56
Single-hop (50 ms) Multi-hop (5 s)
Success [%]
Success [%] Unreliable IP 8.0 97 0.03 57 Reliable IP 305.0 99 3.83 78 I-Not 7.2 100 1.98 68 NDN & HoPP 13.6 100 0.60 98
Protocol
Protocol Unreliable IP 30.6 CoAP PUT (n) 0.2 CoAP GET (n) Reliable IP 5000.0 CoAP PUT (c) 16.3 CoAP PUT (c) NDN & HoPP 27.9 HoPP 13.7 HoPP
52 / 56
Link traversal vs. shortest path using a 15 s publishing interval
1 2 3 4 5 6 Loss Retransmissions Shortest Path NDN I-Not HoPP MQTT-SN (Q0) MQTT-SN (Q1) 10 20 1 2 3 4 5 6 CoAP GET (n) 10 20 CoAP GET (c) 10 20 Data Packets per Publish [#] CoAP OBS 10 20 CoAP PUT (n) 10 20 CoAP PUT (c) Hop Count
53 / 56
Goodput summaries
5 10 15 20 25 30 10 20 NDN theoretical goodput 5 10 15 20 25 30 HoPP theoretical goodput 5 10 15 20 25 30 I-Not theoretical goodput 5 10 15 20 25 30 MQTT-SN (Q0) theoretical goodput 5 10 15 20 25 30 MQTT-SN (Q1) theoretical goodput 5 10 15 20 25 30 10 20 CoAP GET (n) theoretical goodput 5 10 15 20 25 30 CoAP GET (c) theoretical goodput 5 10 15 20 25 30 CoAP OBS theoretical goodput 5 10 15 20 25 30 CoAP PUT (n) theoretical goodput 5 10 15 20 25 30 CoAP PUT (c) theoretical goodput Goodput [KiB / minutes] Content Publishing Intervall [seconds] 54 / 56
Goodput summaries for reliable protocols
5 10 15 20 25 30 5 10 15 20 NDN theoretical goodput 5 10 15 20 25 30 CoAP GET (c) theoretical goodput 5 10 15 20 25 30 MQTT-SN (Q1) theoretical goodput Goodput [KiB / minutes] Content Publishing Intervall [seconds]
55 / 56
Takeaways
▶ All protocols are challenged by the constrained wireless IoT
▶ Reliable transfer oten fails ▶ End-to-end acknowledgments stress lossy links
▶ NDN and HoPP are most robust and stable
▶ Hop-wise caching enhances transport effjciency ▶ NDN susceptible to PIT state overflow
Future Work
▶ Analyze flow control aspects with controlled cross-traffjc ▶ Deploy protocols with full security features enabled
56 / 56