Facing the Challenges of Updating Complex Systems Putting it all - - PowerPoint PPT Presentation

▶
facing the challenges of updating complex systems
SMART_READER_LITE
LIVE PREVIEW

Facing the Challenges of Updating Complex Systems Putting it all - - PowerPoint PPT Presentation

Mar 30, 2024 228 likes •556 views

Facing the Challenges of Updating Complex Systems Putting it all together FOSDEM 2018, Enrico Jrns, Pengutronix e.K. 1/31 About Me Enrico Jrns Embedded Software Engineer RAUC update framework co-maintainer 2/31

slide-1
SLIDE 1 1/31

Facing the Challenges of Updating Complex Systems

Putting it all together

FOSDEM 2018, Enrico Jörns, Pengutronix e.K.
slide-2
SLIDE 2 2/31

About Me

  • Enrico Jörns
  • Embedded Software Engineer
  • RAUC update framework co-maintainer

 

slide-3
SLIDE 3 3/31

Motivation

bootloader application

 

data init watchdog device Updating is a solved topic!..?
slide-4
SLIDE 4 4/31

Motivation

bootloader application deployment server testing

 

data init watchdog device
slide-5
SLIDE 5 6/31

Bootloader Support – Barebox

algorithm confjg persistent status bootchooser

boot targets

  

disk0.1

disk0.2

select
slide-6
SLIDE 6 7/31

Bootchooser Framework

disk0.1 disk0.2 priority: 20 attempts: 3 system0 system1 priority: 10 attempts: 3 reset: power-on system0: disk0.1 system1: disk0.2 highest priority attempts > 0 attempts > 0 attempts -- boot failed watchdog reset power on attempts:=3
slide-7
SLIDE 7 8/31

X86 – Pure UEFI Boot

kernel A kernel B rootfs A rootfs B

HD(1,GPT,<UUID-1>/File(KernelA),rootfs=.... HD(1,GPT,<UUID-2>/File(KernelA),rootfs=.... 0001 0002

BootNext BootOrder

0001 0001,0002 system A system B

BootEntries

slide-8
SLIDE 8 10/31

Updating The Bootloader?

boot0 boot1 user system extCSD

boot.img

eMMC

slide-9
SLIDE 9 11/31

Updating The Bootloader?

boot0 boot1 user system extCSD eMMC

boot.img
slide-10
SLIDE 10 12/31

Updating The Bootloader?

boot0 user system extCSD

boot.img atomic

eMMC

slide-11
SLIDE 11 18/31

Detecting Freezes – Watchdogs!

ROM Loader

Boot- loader

Kernel System

slide-12
SLIDE 12 19/31

Detecting Freezes – Watchdogs!

ROM Loader

Boot- loader

Kernel System

start

reset Watchdog
slide-13
SLIDE 13 20/31

systemd – Watchdog Multiplexer

HW-Watchdog SW-Watchdogs mux WatchdogSec=10 app1.service WatchdogSec=20 app2.service WathcdogSec=30 app2.service RuntimeWatchdogSec=10 ShutdownWatchdogSec=300 watchdog.conf
slide-14
SLIDE 14 21/31

systemd

  • Central control and overview!
  • Service Failure Confjguration
– Restart – RestartSec – ...
  • Watchdog Multiplexer
  • /system-update
– bootstrapping confjg / data
slide-15
SLIDE 15 22/31

Data Storage / Migration

Data in rootfs /etc rootfs rootfs update 1. 2.
  • copy by updater!
  • migration: simple
  • fallback: old data!
slide-16
SLIDE 16 23/31

Data Storage / Migration

Data in separate slot rootfs rootfs update /data /data datafs
  • no copying
  • mount to /data
  • migration: simple
  • fallback: tricky!
slide-17
SLIDE 17 24/31

Data Storage / Migration

rootfs rootfs update /data /data datafs datafs app Data in two separate slots
  • copying by updater
  • migration by

application

  • mounting: tricky
  • fallback: old data!
slide-18
SLIDE 18 25/31

Updating and Trusted Boot

Build System install r/o ext4 block device r/w extract journal tags dm-integrity hash tree image tar Target dm-verity dm-integrity
slide-19
SLIDE 19 26/31

Testing Updates – Labgrid

BareboxDriver ShellDriver Barebox Linux Bundle HW / Qemu Power PowerDriver update-test.py
  • provide update
  • trigger install
  • power cycle
  • test bootloader
  • test linux
Labgrid
slide-20
SLIDE 20 29/31

casync

  • Image updates over Network
– Too large (slow connection) – Temporary storage required → delta updates → not reinvent the wheel “casync (content-addressable synchronisation) is a Linux software utility designed to distribute frequently-updated fjle system images over the Internet.“ [Wikipedia]
slide-21
SLIDE 21 30/31

casync – Chunking

block device / directory tree serialized stream chunk store .caidx #ba32 #007c #2ef5 #7f2b #4a23 #b389 hashing (ID) index fjle compressing
slide-22
SLIDE 22 31/31

casync – Extracting

serialized stream chunk store .caidx #ba32 #007c #2ef5 #7f2b #4a23 #b389 index fjle block device / directory tree https
slide-23
SLIDE 23 32/31

casync – RAUC

chunk store update .caidx metadata slot A seed store slot B

install

slide-24
SLIDE 24 33/31

Field Deployment

  

    

    

update 

  
slide-25
SLIDE 25 34/31

HawkBit – Deployment Server

Web UI Management API Device Integration API

 

  

slide-26
SLIDE 26 35/31

Field Deployment – HawkBit

          

error threshold group 1 group 2 group 3
slide-27
SLIDE 27 36/31

Field Deployment – HawkBit

 

        

error threshold group 1 group 2 group 3
slide-28
SLIDE 28 37/31

Field Deployment – HawkBit

          

error threshold group 1 group 2 group 3   stop!
slide-29
SLIDE 29 38/31

Conclusion

  • Update Frameworks cannot provide full

solutions

  • Not just stacking components
  • Fine-grained confjguration
  • Updating is highly use-case specifjc
slide-30
SLIDE 30 39/31

Questions?

slide-31
SLIDE 31 40/31

Links

  • github.com/rauc
  • rauc.readthedocs.io
  • github.com/systemd/casync
  • github.com/labgrid-project/labgrid
  • github.com/eclipse/hawkbit