Ci L O P t SCI entific Q t application for L earning from O - PowerPoint PPT Presentation

Ci L O P t SCI entific Q t application for L earning from O bservations of P lasmas from S pace D ata Center for Data Sience Paris Saclay Groupe de Travail SPU - Données spatiales - meeting 1er février 2016.

IN THIS PRESENTATION 1. 2. 3.

S UN -E ARTH SYSTEM

C ORONAL MASS EJECTIONS Soleil You are 1-2 days Here

S OLAR WIND

L OTS OF MISSIONS

G RAPHICAL M ULTI & MISSIONS F LEXIBLE C OMMUNITY S MART L earning from S pace Automatic event D ata detection with Machine Learning

TEAM Nico A. « typical user » interface with observers @ LPP H UGO W INTER - Alexis Jeandet Erwan Le Pennec CDD - 12 MOIS Main code designer Expert/consulting expert C++/GUI Main developer Machine Learning GUI Qt Signal Rodrigue Piberne Space Data products / scientific visualization

G RAPHICAL M ULTI & MISSIONS F LEXIBLE C OMMUNITY S MART

W HY M ULTI - MISSIONS ? S AME DATA P LASMA D ENSITY , TEMPERATURE FLOWS P RESSURES D ISTRIBUTIONS E LECTROMAG . E, B, POTENTIAL …

W HY M ULTI - MISSIONS ? SAME DATA FORMAT : CDF e.g. Mission ESA/Cluster, 130TB since 2001 mission NASA/MMS, launched 2015 > 10TB/year

• Multi-mission, intuitive GUI • E ASILY BROWSE DATA PRODUCTS • I NTEROPERABILITY WITH CDPP , NASA, ETC . • LOAD ASCII /CDF FILES … • S IMPLY DRAG PRODUCTS TO PLOT AREA Search data any text will be searched in product meta-data All known products Dynamically filtered data products local, distant…

• R EAD CDF, ASCII… • MISSIONS P LUGINS • AMDA/NASA INTEROPERABILITY G RAPHICAL M ULTI & MISSIONS F LEXIBLE Just get data C OMMUNITY S MART

G RAPHICAL M ULTI & MISSIONS F LEXIBLE C OMMUNITY S MART

E XISTING TOOLS ? S CRIPTING GUI S NO: NO: VERY BAD FOR JUST DATA BROWSING NOT VERY FLEXIBLE R EINVENTING ALL WHEELS TENDENCY FOR « USINES À GAZ » L OTS OF CRAPPY CODE IN NATURE … YES: YES: E ASY DATA BROWSING B RINGS STRONG FLEXIBILITY E ASY ROUTINE TREATMENTS REQUIRED BY RESEARCH E ASY FOR STUDENTS SHARING CODE BATCH ANALYSIS

E XISTING TOOLS ? S CRIPTING GUI S NO: NO: VERY BAD FOR JUST DATA BROWSING NOT VERY FLEXIBLE R EINVENTING ALL WHEELS TENDENCY FOR « USINES À GAZ » L OTS OF CRAPPY CODE IN NATURE … YES: YES: E ASY DATA BROWSING B RINGS STRONG FLEXIBILITY E ASY ROUTINE TREATMENTS REQUIRED BY RESEARCH E ASY FOR STUDENTS SHARING CODE BATCH ANALYSIS

• Technology choices C++ Q T S IMPLE CODE , PORTABLE , PERFORMANCE , GOOD HUGE COMMUNITY COMMUNITY O PEN S OURCE

• Multi-mission, intuitive GUI Scroll and transparently Interactive high download data perf panels easy browsing of data products real time based on update keywords specific toolboxes

• Embedded iPython : power of custom toolkits (homemade or not) • Easy access to user libraries • terminal <—> plots • enable very specific data manipulation (not GUI)

• ELEGANT AND ERGONOMIC • PERFORMANCE AND REAL TIME PLOTTING • INTERACT WITH DATA AND PLOTS • POWER AND FLEXIBILITY OF PYTHON SCRIPTS G RAPHICAL M ULTI & MISSIONS F LEXIBLE C OMMUNITY S MART Visualize complex data interact with data

G RAPHICAL M ULTI & MISSIONS F LEXIBLE C OMMUNITY S MART

« EVENT » Time interval where measures show signatures associated with a physical phenomenon of interest. Usually group them to do statistical studies

• Catalogs of data Gather data for statistics • Catalog = group of data intervals • Data can belong to multiple catalogs • « add to catalog » directly from plot panels • clone/extend features

• Visualizing catalogs Extract and visualize metadata ex : where are all my • Rich automatic metadata intervals located? (user, spacecraft, data products etc.) not just start/stop date and optional description • Easily extract and visualize information from your catalog

• Online community based catalog (> SciQLOP v.1) Improve reproducibility - ANTI-reinventing-the-wheel-tool • Public and group catalogs • Online sharing between all SciQLOP instances • Build catalogs with colleagues

• Catalogs and published studies (> SciQLOP v.1) Improve reproducibility - ANTI-reinventing-the-wheel-tool • Export to publishable additional material catalogs with custom fields • Catalog type = « published event » • Register an event as « published » and add DOI/ paper meta data • SciQLOP will let you know visually that the event you’re looking at has been published and let you easily grab the paper

Share science G RAPHICAL M ULTI & MISSIONS F LEXIBLE • O RGANIZE DATA INTO CATALOGS C OMMUNITY S MART • C OLLABORATIVE CATALOGS • PUSH AND PULL PUBLISHED DATA

G RAPHICAL M ULTI & MISSIONS F LEXIBLE C OMMUNITY S MART L earning from S pace Automatic event D ata detection with Machine Learning

S PACE TIME AMBIGUITY d n i w r a l o s TIME VARYING BOUNDARY CONDITIONS bow shock d n i w r a l o s magnetosheath magnetopause S URFACE WAVES AND PROCESSES

S PACE TIME AMBIGUITY

M ACHINE LEARNING Auto select M’sphere regions M’pause M’sheath Shock Solar wind

Reconnection Visual signatures detection

Trenchi et al. 2008

K ELVIN H ELMHOLTZ

S HOCK CROSSING

C OLLECTING ( AUTOMATICALLY ) DATA IS HARD • DATA IS COMPLEX , NOT REPRODUCIBLE • N AÏVE DETECTION ALGO . BASED ON FIXED RULES GIVE > 70% FALSE DETECTIONS EASIEST THING IS STILL THE EYE • EVERYONE KNOWS THE « TEXTBOOK » EXAMPLE OF OUR FAVORITE PHENOMENA REPRESENTS LESS THAN 1% OF EVENTS • P REVENTS STATISTICAL STUDIES OF PHENOMENA • H OW DO WE USE YEARS OF ARCHIVED DATA ?? • L ISTS ARE COMPILED HERE AND THERE … BAD REPRODUCIBILITY • WHAT DO WE DO WHEN WE RUN OUT OF INTERNS TO SELECT INTERVALS ?

• ML from and for catalogs • Learn from catalogs • suggest new events • scan databases • Extend catalogs • Test performance

• Using catalogs to do science. E.g. shock model as a function of IMF and Sw Mach nber. • What is the 3D shape/position of the shock as a function of solar wind control parameters ? bow shock magnetosheath magnetopause • build and share models based on catalogued data • Export model to 3DView (collaboration with CDPP)

• Using catalogs to do science. e.g. reconnection at the magnetopause • What is the position of the X line on the magnetopause as a function of solar wind control parameters ? magnetopause • build and share models based on catalogued data • Export model to 3DView (collaboration with CDPP)

CDS RAMP I CME A UTOMATIC DETECTION OF ICME S Magnetic clouds: Very geoefficient structure Huge structure lasting typically 1 day cloud Start with a discontinuity : jumps in B, V, n, T than in preceding solar wind sheath Then 2 parts: (1) sheath: large fluctuations (2) Magnetic cloud itself: - smooth variations - Smooth B rotation

Learn data from/for users G RAPHICAL M ULTI & MISSIONS F LEXIBLE • L EARN FROM CATALOGS C OMMUNITY S MART • SUGGEST DATA AND EXTEND CATALOGS • BUILD COMPLEX MODELS FROM DATA

Visualize complex data Just get data interact with data G RAPHICAL M ULTI & MISSIONS F LEXIBLE C OMMUNITY S MART Share science Learn data from/for users

L earning from S pace • D EFINE STRATEGIES TO DETECT : D ata • REGIONS / BOUNDARIES • TAIL / M’ PAUSE / SHOCK / ETC . • SOLAR WIND EVENTS • • SOLAR WIND SHOCKS • M’ PAUSE RECONNECTION • M’ PAUSE KH • I NTEGRATION IN S CI QLOP • LEARN FROM CATALOGS • SCAN DATABASES • SUGGEST EVENTS

FUTURE : Built-in SciQLOP engine?