EISCAT_3D use case of data reuse Ingemar Hggstrm & EISCAT_3D CC - - PowerPoint PPT Presentation

▶

Jan 30, 2023 152 likes •306 views

EISCAT_3D use case of data reuse Ingemar Hggstrm & EISCAT_3D CC team 09/05/2019 1 EISCAT_3D project status Antenna, Receiver, Transmitter units ordered Land acquired (Norway ground preps, Fin/Swe survey) Transmitter control in

SLIDE 1

1 09/05/2019

EISCAT_3D use case of data reuse Ingemar Häggström & EISCAT_3D CC team

SLIDE 2

EISCAT_3D project status Antenna, Receiver, Transmitter units ordered Land acquired (Norway ground preps, Fin/Swe survey) Transmitter control in negotiation Site computing clusters in study (acc: GPU/FPGA)

SLIDE 3

deploy and integrate necessary tools, services and infrastructures

–

Data management and processing

DIRAC interware

–

integration component

–

a single access point towards e-Infrastructures

EUDAT's B2 services

–

unify the data management

–

discovery system across difgerent storages

–

storage access management

EGI and INDIGO services

–

deploying the software stack on HPC/HTC systems including release management

provide secondary services for production operation

–

user authentication and access control

09/05/2019

EISCAT_3D Competence Centre

SLIDE 4

User authentication development

–

EGI Checkin

–

QQ service for CN users in progress

Job submission

–

Deployed on cPouta cloud, CSC

–

Docker containers

–

Existing EISCAT user software (Octave, C)

09/05/2019

EISCAT_3D CC status

Job defjnitjon fjle

[ Executable = "run_rtg_docker.sh"; Arguments = ""; JobName = "my_job_name"; Site = "Cloud.CSC.fi"; CPUTime = 86400; InputSandbox = { "run_rtg_docker.sh", "LFN:/eiscat.se/archive/2016/... } OutputSandbox = {"output/∗"}; ]

SLIDE 5

Dirac fjle catalogue

–

L1 & L2 data

–

B2safe

Processed data

–

L3 data (now only ionospheric data)

–

B2share

–

Search and select data

Processes

–

Dockers/VMs

GUISDAP General analysis L1/L2 → L3
RTG General tool for display of L2 data (L3)

09/05/2019

E3D Services

SLIDE 6

Search & Select data Search & Select data

–

Dirac/B2 fjle catalogue

–

Here: eiscat portal

Select process

–

GUISDAP

Need matlab in VM
gfd window allows

analysis with other parameters

09/05/2019

E3D Process 1

SLIDE 7

Search & Select data Select process

–

RTG

Octave/Matlab
Setup window

09/05/2019

E3D Process 2

SLIDE 8

Plot/Reorder L1/L2 data Sets of data plot windows (movies in batch mode

09/05/2019

E3D RTG Process

EISCAT Tromsø VHF Real Time Graph https://www.eiscat.se/raw/rtg/rtg.cgi?V 1 of 4 24/08/18 09:42

SLIDE 9

9 09/05/2019

E3D RTG Process +

Use RTG semi-fjnished data products (2 entries)

–

RTG only as Reader/Integrator

Provide own defjnition fjle (Can analyse L1 data)

–

Use Data in plot windows (prepared L2 spectral data)

Select window
Provide analysis script (ex Non ionospheric)

SLIDE 10

specfjt.m

–

Needs development

–

May contain several fjles

–

Result is datafjles (or graphics)

09/05/2019

E3D RTG Process Example

%global Ana M0 M1 M2 [dum,d]=strtok(d_ExpInfo); File=fullfile(tempdir,[strtok(d) '_specfit.mat']) if exist(File) load(File) else Results=[]; Time=[]; end GG=3:66; [S,F]=acf2spec(selax.sacf(:,GG),selax.lag'); [NF,NG]=size(S); H=(selax.r0+GG*selax.dt)*.15e6; F=F'; Dl=mean(diff(selax.lag)); Df=mean((diff(F))); %moments M0=sum(S); M1=sum((F*ones(1,NG)).*S); M2=sum(((F*ones(1,NG)-ones(NF,1)*M1).^2).*S); M=[M0' M1' M2']; if 0 Model='fit_gauss'; % Model='fit_lorentz'; M0(find(M0)<0)=eps; M2(find(M2)<0)=eps; X0=[log(M0)' asinh(M1/2)' log(M2)'] O=optimset('Display','off'); end Ana=[]; for i=1:NG if 0 X=fminsearch(Model,X0(i,:),O,selax.sacf(:,GG(1)+ %% Err=norm(feval(Model,X,selax.sacf(:,GG(1)+i-1),F X([1 3])=exp(X([1 3])); X(2)=2*sinh(X(2)); Pars=[X];

SLIDE 11

Setup an EGI Notebook

–

Using the RTG docker SW

Add a small set of data

–

wget from eiscat portal

Insert analysis code

–

First versions

Run RTG interactively

–

Develop/Refjne code

Store analysis script(s)

–

.m/.zip/.tgz

Run batched RTG process

–

CC docker

–

Big data set

09/05/2019

Use Case

SLIDE 12

EGI Marketplace

–

Order octave

Python/Julia/Octave/R

–

T erminal → git, wget,...

09/05/2019

JupyterLab - Octave

SLIDE 13

Setup an EGI Notebook

–

Using the RTG docker

Add a small set of data

–

wget from eiscat portal

Insert analysis script(s)

–

First versions

Run RTG interactively

–

Refjne script(s)

Store analysis script(s)

–

.m/.zip/.tgz

Run batched RTG process

–

Big data set

09/05/2019

JupyterLab - Octave

SLIDE 14