DIY IoT Backend Voon Siong WONG greenfields greenfields 4 years - - PowerPoint PPT Presentation

DIY IoT Backend

Voon Siong WONG

greenfields

greenfields

4 years ago, wanted IoT platform no clear requirements grew organically flaws in hindsight never in production, never fixed hobby project

revisited the project

made list of good, bad, and how it can be better development ~2 weeks, ~800 LOC Ruby runs on 1x t2.small instance response time average 50ms, 95% < 80ms learnt from a lot of bad experiences

disclaimer

not necessarily advocating DIY

• utsourcing to 3rd party vendors can be cheaper

depends on in-house capabilities

Let's Get Started

item 1: time series

don't DIY time series v1 had to do it, not many open source v2 used open source database, threw out a ton of code

read/write optimisation is hard use 1-D sort for index optimisation usually time as primary index

• verwrite or append for same timestamp?

aggregating across multiple tables? geo-spatial capabilities

• eg. measurements in this area at this time

4-D "sort"

we tried building one based on HBase careful planning of Row-key & Column-key

• nly supported overwrite, no appends

fixed time intervals aggregation were scheduled tasks, rather than queries poor performance

unless your job is to build databases, don't

• utside of your control if you're using vendor

item 2: authentication

treat browsers and iot devices equally as first class citizens accessible with fixed or variable API keys

• ptionally with username/password

avoid client SSL cumbersome to generate/manage impractical to install on browsers thus requires proxy server use API Keys fixed API keys variable API key, HMAC request and time deploy with environment variables interactive logins username/password

• auth

item 3: expect change

plan for it very relevant when DIY but use as criteria for judging vendors many vendors are still young

ensure APIs are versioned using path, headers, request params, etc. deprecations devices don't / can't understand warning messages warn developers directly return success on dead APIs don't care about their behaviour any more avoid old devices's retry logic from hammering it

API ownership

APIs can make or break your application

• ne of strongest reasons to DIY

don't forget that it changes!

Item 4: model associations

don't enforce a schema for model relationships embrace schema-less, because sensors are

schema / syntactic fixed associations, inflexible need to know "position" in hierarchy / grouping conflicts when reusing names schema-less / semantic uniqueness by credentials and name hierarchies or grouping stored in text, eg. "state/suburb/street", arbitrarily delimited

item 5: data flows

trickles from device to server streams from server to advanced clients

use regular session-less HTTP for small volumes for greater volume, use a pub-sub mechanism , subscribe to desired channels modern clients use (uni-directional) websockets, resume on connection failures

• lder clients fallback to long-polling

firehose server

item *: nuts & bolts

measurement units

• verwrite vs append

don't require SNI prefer time since epoch text over binary

IoT Backend: Solved Problem?

IoT for hire

authentication, data aggregation, remote control, etc. for hire still maturing sector...

focus on value

this list apply whether you're DIY backend, or evaluating vendors not suggesting one or the other, that's up to you suggest you think about value, rather than technical capability I can, should I?

worth it?

in our case, DIY was worth it (re)development effort was small lessons from previous generation in-house knowledge on how to fix things

LiveVu

Realtime energy monitoring

SmokeAlert

Air Quality Monitoring

disclaimer (repeat)

not necessarily advocating DIY

• utsourcing to 3rd party vendors can be cheaper

depends on in-house capabilities

Thanks!