Ci L O P t SCI entific Q t application for L earning from O - - PowerPoint PPT Presentation

Ci L P O

t

SCIentific Qt application for Learning from Observations of Plasmas

Center for Data Sience Paris Saclay

Groupe de Travail SPU - Données spatiales - meeting 1er février 2016.

Space

from

Data

IN THIS PRESENTATION

1. 2. 3.

SUN-EARTH SYSTEM

CORONAL MASS EJECTIONS

You are Here

Soleil

1-2 days

SOLAR WIND

LOTS OF MISSIONS

MULTI

MISSIONS

GRAPHICAL & FLEXIBLE COMMUNITY SMART

Learning Space

from

Data

Automatic event detection with Machine Learning

HUGO WINTER - CDD - 12 MOIS

TEAM

Erwan Le Pennec Nico A. Alexis Jeandet Rodrigue Piberne

« typical user » interface with observers @ LPP Main code designer expert C++/GUI Space Data products / scientific visualization Expert/consulting Machine Learning Main developer GUI Qt Signal

MULTI

MISSIONS

GRAPHICAL & FLEXIBLE COMMUNITY SMART

PLASMA

DENSITY, TEMPERATURE

FLOWS

PRESSURES

ELECTROMAG.

DISTRIBUTIONS E, B, POTENTIAL… SAME DATA

WHY MULTI-MISSIONS ?

SAME DATA FORMAT : CDF

WHY MULTI-MISSIONS ?

e.g. Mission ESA/Cluster, 130TB since 2001 mission NASA/MMS, launched 2015 > 10TB/year

• Multi-mission, intuitive GUI

All known products Dynamically filtered data products local, distant… Search data any text will be searched in product meta-data

• EASILY BROWSE DATA PRODUCTS
• INTEROPERABILITY WITH CDPP

, NASA, ETC.

• LOAD ASCII/CDF FILES…
• SIMPLY DRAG PRODUCTS TO PLOT AREA

• READ CDF, ASCII…
• MISSIONS PLUGINS
• AMDA/NASA INTEROPERABILITY

Just get data

MULTI

MISSIONS

GRAPHICAL & FLEXIBLE COMMUNITY SMART

MULTI

MISSIONS

GRAPHICAL & FLEXIBLE COMMUNITY SMART

EXISTING TOOLS?

SCRIPTING GUIS

VERY BAD FOR JUST DATA BROWSING REINVENTING ALL WHEELS LOTS OF CRAPPY CODE IN NATURE… BRINGS STRONG FLEXIBILITY

REQUIRED BY RESEARCH SHARING CODE BATCH ANALYSIS

NO: YES:

NOT VERY FLEXIBLE TENDENCY FOR « USINES À GAZ »

EASY DATA BROWSING EASY ROUTINE TREATMENTS EASY FOR STUDENTS

NO: YES:

SCRIPTING GUIS

VERY BAD FOR JUST DATA BROWSING REINVENTING ALL WHEELS LOTS OF CRAPPY CODE IN NATURE… BRINGS STRONG FLEXIBILITY

REQUIRED BY RESEARCH SHARING CODE BATCH ANALYSIS

NO: YES:

NOT VERY FLEXIBLE TENDENCY FOR « USINES À GAZ »

EASY DATA BROWSING EASY ROUTINE TREATMENTS EASY FOR STUDENTS

NO: YES:

EXISTING TOOLS?

• Technology choices

C++ QT

PERFORMANCE, GOOD COMMUNITY

SIMPLE CODE,

PORTABLE, HUGE COMMUNITY

OPEN SOURCE

• Multi-mission, intuitive GUI

easy browsing of data products

Interactive high perf panels

based on keywords specific toolboxes real time update

Scroll and transparently download data

• Embedded iPython : power of custom toolkits

(homemade or not)

• Easy access to user libraries
• terminal <—> plots
• enable very specific data

manipulation (not GUI)

Visualize complex data interact with data

• ELEGANT AND ERGONOMIC
• PERFORMANCE AND REAL TIME PLOTTING
• INTERACT WITH DATA AND PLOTS
• POWER AND FLEXIBILITY OF PYTHON SCRIPTS

MULTI

MISSIONS

GRAPHICAL & FLEXIBLE COMMUNITY SMART

MULTI

MISSIONS

GRAPHICAL & FLEXIBLE COMMUNITY SMART

« EVENT »

Time interval where measures show signatures associated with a physical phenomenon of

• interest. Usually group them to do statistical studies

• Catalogs of data
• Catalog = group
• f data intervals
• « add to catalog »

directly from plot panels

• Data can belong to

multiple catalogs

• clone/extend features

Gather data for statistics

• Visualizing catalogs

Extract and visualize metadata

• Rich automatic metadata

(user, spacecraft, data products etc.) not just start/stop date and

• ptional description
• Easily extract and

visualize information from your catalog

ex : where are all my intervals located?

• Online community based catalog
• Online sharing between

all SciQLOP instances

• Build catalogs with

colleagues

• Public and group

catalogs

(> SciQLOP v.1) Improve reproducibility - ANTI-reinventing-the-wheel-tool

• Catalogs and published studies
• Export to publishable

additional material catalogs with custom fields Improve reproducibility - ANTI-reinventing-the-wheel-tool

• Catalog type = « published

event »

• Register an event as

« published » and add DOI/ paper meta data

• SciQLOP will let you know

visually that the event you’re looking at has been published and let you easily grab the paper (> SciQLOP v.1)

MULTI

MISSIONS

GRAPHICAL & FLEXIBLE COMMUNITY SMART

• ORGANIZE DATA INTO CATALOGS
• COLLABORATIVE CATALOGS
• PUSH AND PULL PUBLISHED DATA

Share science

MULTI

MISSIONS

GRAPHICAL & FLEXIBLE COMMUNITY SMART

Learning Space

from

Data

Automatic event detection with Machine Learning

magnetopause bow shock magnetosheath s

• l

a r w i n d s

• l

a r w i n d

SPACE TIME AMBIGUITY SURFACE WAVES AND

PROCESSES TIME VARYING BOUNDARY CONDITIONS

SPACE TIME AMBIGUITY

MACHINE LEARNING

M’sphere M’pause M’sheath Shock Solar wind

Auto select regions

Reconnection signatures Visual detection

Trenchi et al. 2008

KELVIN HELMHOLTZ

SHOCK CROSSING

COLLECTING (AUTOMATICALLY) DATA IS HARD

EASIEST THING IS STILL THE EYE

• PREVENTS STATISTICAL STUDIES OF PHENOMENA
• HOW DO WE USE YEARS OF ARCHIVED DATA??
• WHAT DO WE DO WHEN WE RUN OUT OF INTERNS TO SELECT INTERVALS?
• LISTS ARE COMPILED HERE AND THERE … BAD REPRODUCIBILITY
• DATA IS COMPLEX, NOT REPRODUCIBLE
• EVERYONE KNOWS THE « TEXTBOOK » EXAMPLE OF OUR FAVORITE PHENOMENA REPRESENTS

LESS THAN 1% OF EVENTS

• NAÏVE DETECTION ALGO. BASED ON FIXED RULES GIVE > 70% FALSE DETECTIONS

• ML from and for catalogs
• Learn from catalogs
• scan databases
• suggest new events
• Extend catalogs
• Test performance

• Using catalogs to do science. E.g. shock

model as a function of IMF and Sw Mach nber.

magnetopause bow shock magnetosheath

• build and share models based
• n catalogued data
• What is the 3D shape/position of the shock as

a function of solar wind control parameters ?

• Export model to 3DView

(collaboration with CDPP)

• Using catalogs to do science. e.g.

reconnection at the magnetopause

magnetopause

• build and share models based
• n catalogued data
• What is the position of the X line on the magnetopause as

a function of solar wind control parameters ?

• Export model to 3DView

(collaboration with CDPP)

CDS RAMP ICME

sheath cloud

Magnetic clouds: Very geoefficient structure Huge structure lasting typically 1 day Start with a discontinuity : jumps in B, V, n, T than in preceding solar wind Then 2 parts: (1) sheath: large fluctuations (2) Magnetic cloud itself:

• smooth variations
• Smooth B rotation

AUTOMATIC DETECTION OF ICMES

MULTI

MISSIONS

GRAPHICAL & FLEXIBLE COMMUNITY SMART

• LEARN FROM CATALOGS
• SUGGEST DATA AND EXTEND CATALOGS
• BUILD COMPLEX MODELS FROM DATA

Learn data from/for users

SMART COMMUNITY GRAPHICAL & FLEXIBLE MULTI

MISSIONS

Learn data from/for users Share science Visualize complex data interact with data Just get data

Learning Space

from

Data

• DEFINE STRATEGIES TO DETECT :
• REGIONS/BOUNDARIES
• TAIL / M’PAUSE / SHOCK / ETC.
• SOLAR WIND
• EVENTS
• SOLAR WIND SHOCKS
• M’PAUSE RECONNECTION
• M’PAUSE KH
• INTEGRATION IN SCIQLOP
• LEARN FROM CATALOGS
• SCAN DATABASES
• SUGGEST EVENTS

Built-in SciQLOP engine? FUTURE :