Photobioreactor system case-study Tom a s Stan ek Tom a s Stan - PowerPoint PPT Presentation

Photobioreactor system case-study Tom´ aˇ s Stanˇ ek Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Overal description Various devices with some autonomous logic and state thermoregulation gas mixture / pressure control air/fluid pumping valve control illumination sensor readings various measurements coordinated by a master controller (PC) performing some higher level logic (e.g. running a biological experiment for a week). Topology is generally master/slave or client/server Performance is not critical. Real-time requirements are confined to each device separately. Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Problem description (laboratory bioreactor) Several bioreactors (along with other instruments) of different configurations are connected to the controller by the user. Bioreactors may be paired with other instruments, but from the users point of view, bioreactors are used separately. The configuration is not known in advance. Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Problem description (laboratory bioreactor) Example of a simple operation Fluorescence measurement (incl. OD): Bubbles are interrupted by a solenoid valve by the bioreactor. Lights are switched off by the bioreactor. Measurements are taken by the bioreactor. Lights back on, bubbles back on. Culture dilution based on optical density: When OD reaches above some predefined level: Bioreactor starts a peristaltic pump with a medium. When OD drops below some predefined level: Bioreactor stops the pump. The whole process may be implemented in the bioreactor firmware. Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Problem description (laboratory bioreactor) Example of a simple operation (bioreactor with a gasanalysing) Fluorescence measurements: Bubbles are interupted by a solenoid valve by the gasanalysing. Lights are switched off by the bioreactor. Measurements are taken by the bioreactor. Lights back on, bubbles back on. Culture dilution based on optical density: When OD reaches above some predefined level: Valves in gasanalysing are set to block gas outflow from the bioreactor. A pump is started via the bioreactor. ... The whole process involves two independent devices and needs to be controlled by the PC. Bad control of the valves / pumps may lead to the flooding of the gasanalysing. Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Problem description (large scale bioreactor) One (or more) bioreactors with additional instruments form a single system. The configuration of the whole system is known in advance. The logic of separate components is much more intertwined, i.e. for almost every action, several components are involved. Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Problem description (large scale bioreactor) Example of an operation Single fluorescence measurement: Bubbles are interrupted via one device as a sequence of valve switches. Lights are temporarily switched off by another one. Measurements are taken by a third. Lights are resumed. Bubbles are resumed. Culture dilution based on optical density: Inflow and outflow pumps alternate between pumping a medium into the tank and pumping out any extra liquid based on reading of a level sensor. When medium runs out, peristaltic pumps are used to mix a new medium from various concetrates. Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Problem description (large scale bioreactor) Bad control of the pumps may destroy the whole tank. Bad control of the gas valves may destroy the tank. Some sensors are sensitive to pressure changes ⇒ gas valve control needs to be fine-tuned. Bad control of the gas valves may even destroy the probe. Some parts of the system are under pressure (4bar, 8bar, ...) Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Problem description (more details) The final configuration may not be known. Devices are connected and disconnected by the user. Communication with devices fails. Electrical spike may reset a usb hub, or a serial dongle. Generally the problem is in receiving only part of a message, not in noise. Devices may turn off all interrupts and go silent/deaf for a few miliseconds or a few seconds as a part of normal operation. Because the device performs some operation with hard realtime requirements. This may happen without any request from the controller (user pushes a button on the device, device does some action for itself). Devices fail/reset (and they do so very quickly). The main controller may fail (deadlock, other software bug). Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Problem description (more details) The development of such systems always has more specifics. Noise on sensors. Generally strange and often non-trivial behavior of various sensors. (temperature dependencies, pressure dependencies) Electrical problems (switching a valve may produce an electrical spike and alter the output of a sensor). Nonlinearities of AD converters. All of those are often quite hard to spot, and almost impossible to identify under a normal operation. Some may be bugs, some may require some sort of calibration and software compensation. Various problems are discovered over time, software/firmware update is required, but devices are all over the world and sending the equipment back to the company is usually (very) inconvenient. Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Problem description (more details) Debugging and testing is problematic. A biological system may take hours to respond. A faulty situation which needs to be handled by the software may be hard to replicate. Hardware may not be ready (yet). And of course a classical environment of a development of any kind. Don’t ever expect the problem to be formulated this way. “Everything is simple”, problems are dismissed or ignored. Everything had to be done yesterday. Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Solution description (basic building blocks) System is built as a composition of various components with uniform communication. Single protocol for higher level communication (messages over channels) Single protocol for communication with devices (messages over point-to-point links). Primitives for handling asynchronous events. Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Small bioreactor schema GUI (Java) I 2 C P 1 P 2 TCP BR 1 BR 2 GAS 1 RS232 Main controller Atmel / ARM PC (Linux) C Various daemons (Mostly Scheme) USB SDB Single protocol for RS232 communication (1) Single protocol for TCP/Unix socket communication (2) SDB (Serial Device Bus) monitors all connected serial ports and brings 1 to the realm of 2 Flat topology is a problem (BR i may be connected to GAS j but there is no way to tell - user needs to pair devices from the GUI). Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Large bioreactor schema TCP µ Linux µ SDB ... dev 1 dev 2 RS232 TCP xport-proxy µ SDB substitutes SDB for a single device. xport-proxy substitutes SDB for internal daemons. Flat topology again (xport-proxy needs to be preconfigured). Different GUI for the touchscreen panel and for the remote access. Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

High level communication protocol Messages over channels. In contrast to messages addressed to agents (Erlang): In the client/server scenario, the client (and its subcomponents) don’t need to be addressable Communication over single channel may be stateful. Component failure results in the closing of a channel. Channels must be opened and closed explicitly. A message over a single channel may need to be routed via several components. Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

Message abstract syntax (S-expressions) e ::= (string) "foo" | (symbol) bar | 42 (integer) | 3.14 (float) | #t | #f (boolean) | ( e e ... ) (list) | (@@ (:slot 1 e ) (:slot 2 e ) ... ) (structure) Structures are non-standard. Pretty much like JSON. All higher level protocols (like rpc) are implemented by s-expressions. Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

S-expressions - example (rpc) (measure-bubble-free-batch red) (ok (@@ (:time 302.2535) (:light-specific (@@ (:red (@@ (:Ft 727.5) (:QY 0.35) (:Fm 1167))) (:blue (@@ (:Ft 2248.36))))) (:light-unspecific (@@ (:temperature 21.1) (:ph 8.7) (:OD-680 0.98) (:OD-735 0.54) (:O2 17.0)))) Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

S-expressions - example (rpc) (let loop ((n 0)) (send channel ‘(add 5)) (match (recv channel) (print (recv channel)) ((’get) ;; (ok) (send channel ‘(ok ,n)) (loop n)) (send channel ‘(add 37)) (print (recv channel)) ((’add m) ;; (ok) (send channel ‘(ok)) (loop (+ n m))) (send channel ‘(get)) (print (recv channel)) ((’shutdown) ;; (ok 42) (send channel ‘(ok)) (close-channel channel)) (else (send channel ‘(error "invalid request")) (loop n)))) Tom´ aˇ s Stanˇ ek Photobioreactor system case-study

S-expressions TCP µ Linux µ SDB ... dev 1 dev 2 RS232 TCP xport-proxy All communication along blue lines uses binary encoded s-expressions. (For either rpc or synchronization). Binary encoding allows extensions, like image data, etc. (No such extensions were used in the bioreactor system though.) Tom´ aˇ s Stanˇ ek Photobioreactor system case-study