FlyLoop: A Micro Framework for Rapid Development of Physiological - - PowerPoint PPT Presentation

▶
flyloop a micro framework for rapid development of
SMART_READER_LITE
LIVE PREVIEW

FlyLoop: A Micro Framework for Rapid Development of Physiological - - PowerPoint PPT Presentation

Jan 13, 2023 102 likes •441 views

FlyLoop: A Micro Framework for Rapid Development of Physiological Computing Systems Evan M. Peck , Eleanor Easse, Nick Marshall, William Stratton, and L. Felipe Perrone Department of Computer Science Bucknell University, PA, U.S.A. 25. June

slide-1
SLIDE 1
  • 25. June 2015

EICS 2015, Duisburg, Germany

FlyLoop: A Micro Framework for Rapid Development of Physiological Computing Systems

1

Evan M. Peck, Eleanor Easse, Nick Marshall, William Stratton, and L. Felipe Perrone Department of Computer Science Bucknell University, PA, U.S.A.

slide-2
SLIDE 2
  • 25. June 2015

EICS 2015, Duisburg, Germany

Physiological Computing Systems

2

“Research on adaptive systems, especially in the area of affective computing, places enormous emphasis on the capacity of the machine to monitor and make accurate inferences about the psychological state of the user.” S.H. Fairclough 2015

http://physiologicalcomputing.org/2015/03/we-need-to-talk-about-clippy/

slide-3
SLIDE 3
  • 25. June 2015

EICS 2015, Duisburg, Germany

Wish List

  • Adaptive systems that can sense the psychological

state of the user and can react intelligently to physiological input.

  • Easily built custom physiological computing systems for

rapid prototyping, replication, and validation.

3

slide-4
SLIDE 4
  • 25. June 2015

EICS 2015, Duisburg, Germany

The Biocybernetic Loop

4

SENSE SENSE

. . .

EXTRACT FEATURES + MAP TO USER SATE

CALIBRATE

REAL TIME OUTPUT MODIFY

slide-5
SLIDE 5
  • 25. June 2015

EICS 2015, Duisburg, Germany

MUSE Brain Sensing Headband

5

slide-6
SLIDE 6
  • 25. June 2015

EICS 2015, Duisburg, Germany

Tobii EyeX Controller

6

slide-7
SLIDE 7
  • 25. June 2015

EICS 2015, Duisburg, Germany

Apple Watch

7

slide-8
SLIDE 8
  • 25. June 2015

EICS 2015, Duisburg, Germany

Challenges

  • Build custom systems for proof of concept

investigations and rapid prototyping. Easy to make them inflexible.

  • Support any number of data sources with varying

velocity and modality.

  • Couplings between components of the pipeline can be

unwieldy.

8

slide-9
SLIDE 9
  • 25. June 2015

EICS 2015, Duisburg, Germany

Observer Design Pattern (Gamma et al. 1995)

9

SUBJECT OBSERVER

request notification

slide-10
SLIDE 10
  • 25. June 2015

EICS 2015, Duisburg, Germany

Observer Design Pattern (Gamma et al. 1995)

10

SUBJECT OBSERVER

notification Push Model

slide-11
SLIDE 11
  • 25. June 2015

EICS 2015, Duisburg, Germany

Observer Design Pattern (Gamma et al. 1995)

11

SUBJECT OBSERVER

notification Push Model pusher receiver

slide-12
SLIDE 12
  • 25. June 2015

EICS 2015, Duisburg, Germany

Observer Design Pattern (Gamma et al. 1995)

12

SUBJECT OBSERVER OBSERVER OBSERVER

. . . notifications Push Model pusher receiver

slide-13
SLIDE 13
  • 25. June 2015

EICS 2015, Duisburg, Germany

FlyLoop

A lightweight, minimal programming framework whose goal is usability for the developer. FlyLoop is a Java microframework with foundation classes and interfaces based on a data flow model.

13

slide-14
SLIDE 14
  • 25. June 2015

EICS 2015, Duisburg, Germany

FlyLoop Core Modules

14

  • Wrapping sensor specific data streams
  • Manipulation of data streams
  • Mapping of data to user state
  • Marshall output to specific needs
slide-15
SLIDE 15
  • 25. June 2015

EICS 2015, Duisburg, Germany

FlyLoop Core Modules

15

  • Modules inherit from a Receiver class and/or

implement a Pusher interface

  • Data is transferred in or out according to a

system-wide polling rate

  • Data is passed around as a Java Object
slide-16
SLIDE 16
  • 25. June 2015

EICS 2015, Duisburg, Germany

DataSource

16

  • startCollection : activates component
  • getOutput : returns single data point
  • push : sends data point to receivers

Interfaces with any kind of streaming sensor; makes no assumptions about incoming data A few core operations:

slide-17
SLIDE 17
  • 25. June 2015

EICS 2015, Duisburg, Germany

Filter

17

  • getDataPoints : ask for a window of data of

arbitrary size Applies some transformation

  • n data stream

A core operation: Designed to allow developer to focus on signal processing algorithm rather than on timing of data transfer

slide-18
SLIDE 18
  • 25. June 2015

EICS 2015, Duisburg, Germany

Calibrator

18

Small set of built-in functions to communicate labels on training data to Learner and to determine mode of operation (training and predicting) Encapsulates training tasks.

slide-19
SLIDE 19
  • 25. June 2015

EICS 2015, Duisburg, Germany

Learner

19

Takes input from any DataSource

  • r Filter; builds model based on

training data or outputs real-time classification based on new data A lot of complexity is encapsulated in this module (statistics, machine learning).

  • learn : ask for a window of data of arbitrary

size A core operation:

slide-20
SLIDE 20
  • 25. June 2015

EICS 2015, Duisburg, Germany

Output

20

Any component in the pipeline can push to it. This generic component marshals data into different formats for different purposes. For instance:

  • Create logs,
  • Interface with data visualization tools,
  • Transmit model classifications over the network,
  • etc.
slide-21
SLIDE 21
  • 25. June 2015

EICS 2015, Duisburg, Germany

Synchronization

When a component takes input from multiple DataSources, sampling rates may not match.

21

A sample is available from one source, but not from

  • another. Possible behaviors:

(1) Repeat the previous input when there is no new data. (2) Push null values when there is no new data.

slide-22
SLIDE 22
  • 25. June 2015

EICS 2015, Duisburg, Germany

Simple Example

22

Mouse Coords Smooth SVM Console

slide-23
SLIDE 23
  • 25. June 2015

EICS 2015, Duisburg, Germany

Complex Example

23

Mouse Coords Smooth SVM Brain Sensor 1 Brain Sensor 2 Low Pass Filter Low Pass Filter Mean Ratio Adaptive App Brain Features Log File Raw Brain Data Log File

slide-24
SLIDE 24
  • 25. June 2015

EICS 2015, Duisburg, Germany

Limitations

  • Filters are constrained to online algorithms
  • Component structure makes it challenging

to create manipulations that require a global view of data

24

slide-25
SLIDE 25
  • 25. June 2015

EICS 2015, Duisburg, Germany

Ongoing and Future Work

  • Refactoring / Cleaning up
  • Public release under MIT license
  • Creating persistent configuration files to

enable replication

  • Creating a configuration language on top
  • f FlyLoop (compiles to Java) - possibly

visual

25

slide-26
SLIDE 26
  • 25. June 2015

EICS 2015, Duisburg, Germany

Thanks for your attention!

Questions?

26

slide-27
SLIDE 27
  • 25. June 2015

EICS 2015, Duisburg, Germany

Pusher (Interface)

public interface Pusher { public Receiver[] getReceivers(); public void setReceivers(Receiver receivers[]); public void setReceivers(Receiver receiver); public void push(); }

27

slide-28
SLIDE 28
  • 25. June 2015

EICS 2015, Duisburg, Germany

Receiver (Abstract Class)

28

public abstract class Receiver {

  • public void addSource(int source);

public void addSources(int[] sources); public ArrayList<Integer> getSourceIDs(); void receive(Object data, int id); public Object[] getData(); public Object getDataPoint(int i); public abstract void processData(); }

slide-29
SLIDE 29
  • 25. June 2015

EICS 2015, Duisburg, Germany

Filter (Abstract Class)

29

public abstract class Filter extends Receiver implements Pusher { private Receiver[] receivers; private Object outputBuffer; private Queue<Object[]> allData; private int interval; public Filter(int interval, int max); public Filter(int interval); public Filter(); public Object[] getInterval(); public Object[] getInterval(int i); public Object[] getInterval(int i, int n); public abstract Object filterData(); public void push(); public Receiver[] getReceivers(); public void setReceivers(Receiver[] receivers); public void setReceivers(Receiver receiver); public void processData(); }

slide-30
SLIDE 30
  • 25. June 2015

EICS 2015, Duisburg, Germany

Calibrator (Abstract Class)

30

public abstract class Calibrator implements Runnable { public Calibrator(String[] states, Learner[] learners, String inFile, String outFile); public Calibrator(String[] states, Learner learner, String inFile, String outFile); public Calibrator(String[] states, Learner learner); public Calibrator(String[] states, Learner[] learners); public void startCalibration(); public abstract void skipCalibrator(); public abstract void initCalibrator(); public void finishCalibrating(); public abstract void calibrate(); }

slide-31
SLIDE 31
  • 25. June 2015

EICS 2015, Duisburg, Germany

Learner (Abstract Class)

31

public abstract class Learner extends Receiver implements Pusher { public Learner(int max, boolean outputConfidence); public void processData(); public String getState(); public boolean isCalibrating(); public void setCalibrating(boolean isCalibrating); public void startCalibrating(); public void stopCalibrating(); public void pauseCalibrating(); public String getCalibrationState(); public void setCalibrationState(String calibrationState); protected void setState(String state); public void push(); public Double getConfidence(); public void setConfidence(Double confidence); public Receiver[] getReceivers(); public void setReceivers(Receiver[] receivers); public void setReceivers(Receiver receiver); public abstract void learn(); }

slide-32
SLIDE 32
  • 25. June 2015

EICS 2015, Duisburg, Germany

Data Source (Abstract Class)

32

public abstract class DataSource implements Pusher { public DataSource(); public DataSource(boolean repeat); public abstract void startCollection(); public void push(); public Receiver[] getReceivers(); public void setReceivers(Receiver[] receivers); public void setReceivers(Receiver receiver); public abstract Object getOutput(); }

slide-33
SLIDE 33
  • 25. June 2015

EICS 2015, Duisburg, Germany

Output (Abstract Class)

33

public abstract class Output extends Receiver {

  • public Output(boolean stateChange, int max);

public void processData(); public abstract void output(); }