Photobioreactor system case-study Tom a s Stan ek Tom a s Stan - - PowerPoint PPT Presentation

Photobioreactor system case-study

Tom´ aˇ s Stanˇ ek

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Overal description

Various devices with some autonomous logic and state

thermoregulation gas mixture / pressure control air/fluid pumping valve control illumination sensor readings various measurements

coordinated by a master controller (PC) performing some higher level logic (e.g. running a biological experiment for a week). Topology is generally master/slave or client/server Performance is not critical. Real-time requirements are confined to each device separately.

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Problem description (laboratory bioreactor)

Several bioreactors (along with other instruments) of different configurations are connected to the controller by the user. Bioreactors may be paired with other instruments, but from the users point of view, bioreactors are used separately. The configuration is not known in advance.

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Problem description (laboratory bioreactor)

Example of a simple operation

Fluorescence measurement (incl. OD):

Bubbles are interrupted by a solenoid valve by the bioreactor. Lights are switched off by the bioreactor. Measurements are taken by the bioreactor. Lights back on, bubbles back on.

Culture dilution based on optical density:

When OD reaches above some predefined level: Bioreactor starts a peristaltic pump with a medium. When OD drops below some predefined level: Bioreactor stops the pump.

The whole process may be implemented in the bioreactor firmware.

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Problem description (laboratory bioreactor)

Example of a simple operation (bioreactor with a gasanalysing)

Fluorescence measurements:

Bubbles are interupted by a solenoid valve by the gasanalysing. Lights are switched off by the bioreactor. Measurements are taken by the bioreactor. Lights back on, bubbles back on.

Culture dilution based on optical density:

When OD reaches above some predefined level: Valves in gasanalysing are set to block gas outflow from the bioreactor. A pump is started via the bioreactor. ...

The whole process involves two independent devices and needs to be controlled by the PC. Bad control of the valves / pumps may lead to the flooding of the gasanalysing.

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Problem description (large scale bioreactor)

One (or more) bioreactors with additional instruments form a single system. The configuration of the whole system is known in advance. The logic of separate components is much more intertwined, i.e. for almost every action, several components are involved.

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Problem description (large scale bioreactor)

Example of an operation

Single fluorescence measurement:

Bubbles are interrupted via one device as a sequence of valve switches. Lights are temporarily switched off by another one. Measurements are taken by a third. Lights are resumed. Bubbles are resumed.

Culture dilution based on optical density:

Inflow and outflow pumps alternate between pumping a medium into the tank and pumping out any extra liquid based on reading of a level sensor. When medium runs out, peristaltic pumps are used to mix a new medium from various concetrates.

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Problem description (large scale bioreactor)

Bad control of the pumps may destroy the whole tank. Bad control of the gas valves may destroy the tank. Some sensors are sensitive to pressure changes ⇒ gas valve control needs to be fine-tuned. Bad control of the gas valves may even destroy the probe. Some parts of the system are under pressure (4bar, 8bar, ...)

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Problem description (more details)

The final configuration may not be known. Devices are connected and disconnected by the user. Communication with devices fails.

Electrical spike may reset a usb hub, or a serial dongle. Generally the problem is in receiving only part of a message, not in noise. Devices may turn off all interrupts and go silent/deaf for a few miliseconds

• r a few seconds as a part of normal operation.

Because the device performs some operation with hard realtime requirements. This may happen without any request from the controller (user pushes a button

• n the device, device does some action for itself).

Devices fail/reset (and they do so very quickly). The main controller may fail (deadlock, other software bug).

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Problem description (more details)

The development of such systems always has more specifics.

Noise on sensors. Generally strange and often non-trivial behavior of various sensors. (temperature dependencies, pressure dependencies) Electrical problems (switching a valve may produce an electrical spike and alter the output of a sensor). Nonlinearities of AD converters.

All of those are often quite hard to spot, and almost impossible to identify under a normal operation. Some may be bugs, some may require some sort of calibration and software compensation. Various problems are discovered over time, software/firmware update is required, but devices are all over the world and sending the equipment back to the company is usually (very) inconvenient.

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Problem description (more details)

Debugging and testing is problematic.

A biological system may take hours to respond. A faulty situation which needs to be handled by the software may be hard to replicate. Hardware may not be ready (yet).

And of course a classical environment of a development of any kind.

Don’t ever expect the problem to be formulated this way. “Everything is simple”, problems are dismissed or ignored. Everything had to be done yesterday.

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Solution description (basic building blocks)

System is built as a composition of various components with uniform communication. Single protocol for higher level communication (messages over channels) Single protocol for communication with devices (messages over point-to-point links). Primitives for handling asynchronous events.

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Small bioreactor schema

BR1 RS232 USB P1 P2 I2C GAS1 BR2 TCP

SDB

Main controller PC (Linux) Various daemons (Mostly Scheme) GUI (Java) Atmel / ARM C

Single protocol for RS232 communication (1) Single protocol for TCP/Unix socket communication (2) SDB (Serial Device Bus) monitors all connected serial ports and brings 1 to the realm of 2 Flat topology is a problem (BRi may be connected to GASj but there is no way to tell - user needs to pair devices from the GUI).

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Large bioreactor schema

dev1 RS232 TCP dev2 ... µLinux µSDB TCP

xport-proxy

µSDB substitutes SDB for a single device. xport-proxy substitutes SDB for internal daemons. Flat topology again (xport-proxy needs to be preconfigured). Different GUI for the touchscreen panel and for the remote access.

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

High level communication protocol

Messages over channels. In contrast to messages addressed to agents (Erlang):

In the client/server scenario, the client (and its subcomponents) don’t need to be addressable Communication over single channel may be stateful. Component failure results in the closing of a channel. Channels must be opened and closed explicitly. A message over a single channel may need to be routed via several components.

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Message abstract syntax (S-expressions)

e ::= "foo" (string) | bar (symbol) | 42 (integer) | 3.14 (float) | #t | #f (boolean) | (e e ...) (list) | (@@ (:slot1 e) (:slot2 e) ...) (structure) Structures are non-standard. Pretty much like JSON. All higher level protocols (like rpc) are implemented by s-expressions.

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

S-expressions - example (rpc)

(measure-bubble-free-batch red) (ok (@@ (:time 302.2535) (:light-specific (@@ (:red (@@ (:Ft 727.5) (:QY 0.35) (:Fm 1167))) (:blue (@@ (:Ft 2248.36))))) (:light-unspecific (@@ (:temperature 21.1) (:ph 8.7) (:OD-680 0.98) (:OD-735 0.54) (:O2 17.0))))

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

S-expressions - example (rpc)

(let loop ((n 0)) (match (recv channel) ((’get) (send channel ‘(ok ,n)) (loop n)) ((’add m) (send channel ‘(ok)) (loop (+ n m))) ((’shutdown) (send channel ‘(ok)) (close-channel channel)) (else (send channel ‘(error "invalid request")) (loop n)))) (send channel ‘(add 5)) (print (recv channel)) ;; (ok) (send channel ‘(add 37)) (print (recv channel)) ;; (ok) (send channel ‘(get)) (print (recv channel)) ;; (ok 42)

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

S-expressions

dev1 RS232 TCP dev2 ... µLinux µSDB TCP

xport-proxy

All communication along blue lines uses binary encoded s-expressions. (For either rpc or synchronization). Binary encoding allows extensions, like image data, etc. (No such extensions were used in the bioreactor system though.)

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Low level communication protocol

The resources of the chip in each device are very limited and from the device point of view, the protocol needs to be simple as possible. A device processes at most one external request at a time. It may, however, perform some autonomous logic (pressure or temperature regulation etc.) A device is connected by a single serial line (sometimes I2C) Both requests and responses take form of a packet containing:

Magic number Sequence number Length Payload Crc

Sequence numbers allows detection of lost packets, possibly replays of responses and out of bound messages from the device.

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Low level communication protocol

Payload format

Payload of both response and request consists of:

Tag Data

Format of data for each tag for a specific device is defined in a device manifest. (define-command 1123 (measure-od (:uint8 channel) (:uint16 repeats))) (define-type 2123 (od (:int32 flash) (:int32 background))) Requests in form of s-expressions are understood by serial-device-bus and communicated to a device in a proper format.

→ (measure-od 0 20) ← (od 10000 100)

Manifests needs to be stored on the controller. They could be, in principle,

• btainable from the device itself.

However, they serve a documentation purpose as well.

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Low level communication protocol

Common (hardwired) commands

(ping) (who) (device-id) (master-ping timeout) (get-last-response seq)

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Low level communication protocol

A device may signal (at any time) a temporary busy mode in which all it’s interrupts are off (and therefore the device seems to be disconnected). ← (busy 500) ← (ready) When a device doesn’t resume communication after specified interval, it is considered disconnected. A device may be attached to an arbitrary serial port (its serial port may even change when an electrical spike resets a USB-hub/dongle), but it’s uniquely identified by its device-id. Devices communicate over different serial speeds. All this behavior is somewhat non-trivial, but abstracted by the serial-device-bus.

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

master-ping

A facility to handle controller or communication failures. Very simple but quite useful safety net. Master ping is issued periodically by a higher level daemon to confirm that a device should continue its operation. Handles failures of the controller. Detects device restarts.

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Protocols using s-expressions

Subsystem connection

→ (connect subsystem1 subsystem2) ← (ok) Messages of the channel are now routed to subsystem2 of subsystem1

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Protocols using s-expressions

RPC

→ (request arg1 arg2 ...) ← (ok response) | (ok) | (error "message")

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Protocols using s-expressions

Value observation (scalar)

→ (wait) ← (ok 7.4) - current value → (wait) The other side doesn’t respond until the observed value has changed.

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Protocols using s-expressions

Value observation (map)

→ (wait) ← (ok ((add key1 value1) (add key2 value2) ...)) → (wait) ... ← (ok ((del key1) (add key3 value3) ...))

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Demo

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Consequences

Creation of active composable components. The whole system may be controlled by another piece of software and used as a big component (the controller could easily run on raspberry-pi). Rich remote gui (different guis for different purposes). All devices are controlled in uniform fashion. Prototyping and problem detection in devices (Arbitrary devices may be just plugged in and scripted in a matter of minutes.) Same framework is used in production as well as for prototypes. Calibration scripts. Controlled from different languages (currently Scheme, Java and Matlab)

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Demo2

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Asynchronous primitives

Inspired by ConcurrentML events / Scheme48 rendezvous. A rendezvous is generally a thing, where producer meets a consumer, providing a value. The generic interface is consumer oriented and very simple. The producer specific functionality is built-in into various kinds of rendezvous. Rendezvous are first class objects. A consumer chooses what to wait for and when. A single consumer (thread) may wait for multiple independent events to

• ccur (that is the whole point!).

Several consumers (threads) may wait at the same time for a single event to occur. Rendezvous are safe to discard. (No registered callbacks / listeners / etc. GC will take care of them).

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Asynchronous primitives

Rendezvous

A rendezvous is in one of three states: fresh → ready → synchronized. When a producer generates an event (specific for a given rendezvous), it marks the rendezvous ready. When a consumer waits for an event, which has become ready, it synchronizes it, yielding a value for the consumer. The synchronization code is provided by the producer and often involves some side effect (like popping a value from a channel queue). The synchronization code is executed by the consumer. When a rendezvous is synchronized, the resulting value is memoized inside the rendezvous and subsequent attempts for synchronization will just yeild this value. When several consumers waits on a single (unsynchronized) rendezvous, at most one will synchronize it. The others will just receive the resulting value.

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Asynchronous primitives

Consumer interface

rv-wait : (Rv a) → a rv-wrap : (Rv a) x (a → b) → (Rv b) rv-choice : [(Rv a)] → (Rv a) rv-ready? : (Rv a) → Bool (rv-select rv1 rv2 ...) = (rv-wait (rv-choice rv1 rv2 ...))

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Asynchronous primitives - (rv-wait)

(rv-wait rv) - waits on and then synchronizes rv, returning value yielded by rv. Jar - one-time placeholder. Initially empty. When filled, cannot be emptied.

(let ((jar-1 (make-jar)) (jar-2 (make-jar))) (thread-rv (thread-sleep! 1) (fill-jar! jar-1 "cookie")) (thread-rv (thread-sleep! 2) (fill-jar! jar-2 "coin")) (thread-rv (print (rv-wait jar-1))) ;; prints cookie in 1s (thread-rv (print (rv-wait jar-2))) ;; prints coin in 2s (print (rv-wait jar-2)) ;; prints coin in 2s (print (rv-wait jar-1))) ;; prints cookie in 2s

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Asynchronous primitives - (rv-wrap)

(rv-wrap rv1 f ) = rv2 (rv-ready? rv1) ⇔ (rv-ready? rv2) (rv-wait rv2) = (f (rv-wait rv2)) rv-wait and wrapped rendezvous respect tail-call behavior. Wrapped rendezvous are not memoized. Therefore f could be called by several threads. More often than not, f is used as a continuation. Motivation behind rv-wrap will be more

• bvious with the rv-choice combinator.

(let* ((jar (make-jar)) (rv1 (rv-wrap jar (lambda (x) (print 1) (list 1 x)))) (rv2 (rv-wrap rv1 (lambda (x) (print 2) (list 2 x))))) (thread-rv (thread-sleep! 1) (fill-jar! jar "cookie")) (print (rv-wait rv1)) ;; 1 ;; (w1 "cookie") (print (rv-wait rv2))) ;; 2 ;; 1 ;; (w2 (w1 "cookie"))

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Asynchronous primitives - (rv-choice)

Nondeterministic choice. (rv-choice rv1 rv2) = rv3 (rv-ready? rv3) ⇔ (rv-ready? rv1) ∨ (rv-ready? rv2) (rv-wait rv3) ⇔ (rv-wait rv1) ∨ (rv-wait rv2)

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Asynchronous primitives - (rv-choice example)

(let ((jar1 (make-jar)) (jar2 (make-jar))) (thread-rv (f jar1)) ;; fills jar1 at some point (thread-rv (g jar2)) ;; fills jar2 at some point (print ;; prints either (1 x) or (2 y) (rv-wait (rv-choice (rv-wrap jar1 (lambda (x) (print "jar1 was first with value: " x) (list 1 x))) (rv-wrap jar2 (lambda (y) (print "jar2 was first with value: " y) (list 2 y)))))))

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Some common primitives returning rendezvous

(always-rv value) never-rv (timeout-rv seconds) (recv-rv channel) (rpc-rv channel message) (thread-rv e1 e2 ...) (process-rv "path/to/binary" "arg1" "arg2" ...) ...

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

More convoluted example

(define (agent ch stop-rv pumping-model) (let loop ((pump-off-rv never-rv) (measure-rv (always-rv #t))) (rv-select (rv-wrap (recv-rv ch) (match-lambda ((’set-pump! on/off) (model-put! pumping-model on/off) (send ch ‘(ok)) (loop (if on/off (timeout-rv 5) never-rv) msr-rv)) (else (send ‘(error "invalid command")) (loop pump-off-rv msr-rv)))) (rv-wrap pump-off-rv (lambda ( ) (print "pump timeout: turning off the pump") (model-put! pumping-model #f) (loop never-rv msr-rv))) (rv-wrap measure-rv (lambda ( ) (print "some periodic measurement") (loop pump-off-rv (timeout-rv 2)))) (rv-wrap stop-rv (lambda ( ) (print "shutdown")))))) Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Threads and exceptions

Conventional try/catch doesn’t fit well in the presence of more threads. (Grd a) = Ok a | Error e (grd e1 e2 ...) : (Grd t) where t is the type of the last expression. (ungrd (Ok value)) = value (ungrd (Error e)) = (raise e) (thread-rv e1 e2 ...) : (Rv (Grd t)) where t is the type of the last expression. (rpc-rv channel msg) : (Rv (Grd t)) where t is the type of the response. With pattern matching, Grds are syntactically even more convenient than try/catch

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Threads and exceptions - pmap example

(define (pmap f list) (map ungrd (map rv-wait (map (lambda (x) (thread-rv (f x))) list))))

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Threads and exceptions - rpc to multiple devices example

(let ((ans1-rv (rpc-rv dev1 ‘(request1))) (ans2-rv (rpc-rv dev2 ‘(request2))) (ans3-rv (rpc-rv dev3 ‘(request3)))) (match (rv-wait ans1-rv) ((:ok ans1) ... (match (rv-wait ans2-rv) ((:ok ans2) ...) ((:error e2) ...))) ((:error e1) ...))) Requests are sent and serviced in parallel by different devices. dev1, dev2, dev3 may safely be just one channel. Requests and responses are matched as expected. A client may safely ignore any of the responses. No request/response mismatches. Obviously, an order in which responses are waited for is irrelevant.

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Asynchronous primitives - possible extensions

rv-and : (Rv a) x (Rv b) → (Rv (a x b))

Not very complicated and interesting, but could be useful.

rv-sequence : (Rv a) x (a → (Continue (Rv b) | Return b))) → (Rv b)

Sort of a thread animated by the consumer. Potentially very useful.

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Asynchronous primitives - possible extensions

model-observer o - interface to a remote or a local model (observer-value o) - latest known (synchronized) value of a model (observer-rv o) - rendezvous with next model change Trivial extension: (compound-observer f o1 o2 ...) = co (observer-rv co) = (rv-wrap (rv-choice (observer-rv o1) (observer-rv o2) ...) (lambda ( ) (f (observer-value o1) (observer-value o2) ...)))

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Asynchronous primitives - possible extensions

But with rv-sequence: (compound-condition-rv pred o1 o2 ...) = (let ((co (compound-observer pred o1 o2 ...))) (let next-try () (rv-sequence (observer-rv co) (lambda (holds) (if holds (Return #t) (Continue (next-try))))))) General asynchronous condition Consumers could wait/choose/etc on arbitrarily complicated conditions. Again, the interface doesn’t enforce any callback registration, etc.

There could be some explicitly managed resources thou (channels to remote models etc.)

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Thank you for your attention. Questions?

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study