EISCAT_3D use case of data reuse Ingemar Häggström & EISCAT_3D CC team 09/05/2019 1
EISCAT_3D project status Antenna, Receiver, Transmitter units ordered Land acquired (Norway ground preps, Fin/Swe survey) Transmitter control in negotiation Site computing clusters in study (acc: GPU/FPGA)
EISCAT_3D Competence Centre deploy and integrate necessary tools, services and infrastructures Data management and processing – DIRAC interware integration component – a single access point towards e-Infrastructures – EUDAT's B2 services unify the data management – discovery system across difgerent storages – storage access management – EGI and INDIGO services deploying the software stack on HPC/HTC systems including release management – provide secondary services for production operation user authentication and access control – 09/05/2019 3
EISCAT_3D CC status User authentication development EGI Checkin – QQ service for CN users in progress – Job submission Deployed on cPouta cloud, CSC – Job defjnitjon fjle Docker containers – [ Executable = "run_rtg_docker.sh"; Existing EISCAT user software (Octave, C) – Arguments = ""; JobName = "my_job_name"; Site = "Cloud.CSC.fi"; CPUTime = 86400; InputSandbox = { "run_rtg_docker.sh", "LFN:/eiscat.se/archive/2016/... } OutputSandbox = {"output/∗"}; ] 09/05/2019 4
E3D Services Dirac fjle catalogue L1 & L2 data – B2safe – Processed data L3 data (now only ionospheric data) – B2share – Search and select data – Processes Dockers/VMs – GUISDAP General analysis L1/L2 → L3 ● RTG General tool for display of L2 data (L3) ● 09/05/2019 5
E3D Process 1 Search & Select data Search & Select data Dirac/B2 fjle catalogue – Here: eiscat portal – Select process GUISDAP – Need matlab in VM ● gfd window allows ● analysis with other parameters 09/05/2019 6
E3D Process 2 Search & Select data Select process RTG – Octave/Matlab ● Setup window ● 09/05/2019 7
E3D RTG Process Plot/Reorder L1/L2 data Sets of data plot windows (movies in batch mode EISCAT Tromsø VHF Real Time Graph https://www.eiscat.se/raw/rtg/rtg.cgi?V 1 of 4 24/08/18 09:42 09/05/2019 8
E3D RTG Process + Use RTG semi-fjnished data products (2 entries) RTG only as Reader/Integrator – Provide own defjnition fjle (Can analyse L1 data) ● Use Data in plot windows (prepared L2 spectral data) – Select window ● Provide analysis script (ex Non ionospheric) ● 09/05/2019 9
E3D RTG Process Example %global Ana M0 M1 M2 [dum,d]=strtok(d_ExpInfo); specfjt.m File=fullfile(tempdir,[strtok(d) '_specfit.mat']) if exist(File) load(File) else Needs development – Results=[]; Time=[]; end GG=3:66; – May contain several fjles [S,F]=acf2spec(selax.sacf(:,GG),selax.lag'); [NF,NG]=size(S); H=(selax.r0+GG*selax.dt)*.15e6; – Result is datafjles (or graphics) F=F'; Dl=mean(diff(selax.lag)); Df=mean((diff(F))); %moments M0=sum(S); M1=sum((F*ones(1,NG)).*S); M2=sum(((F*ones(1,NG)-ones(NF,1)*M1).^2).*S); M=[M0' M1' M2']; if 0 Model='fit_gauss'; % Model='fit_lorentz'; M0(find(M0)<0)=eps; M2(find(M2)<0)=eps; X0=[log(M0)' asinh(M1/2)' log(M2)'] O=optimset('Display','off'); end Ana=[]; for i=1:NG if 0 X=fminsearch(Model,X0(i,:),O,selax.sacf(:,GG(1)+ %% Err=norm(feval(Model,X,selax.sacf(:,GG(1)+i-1),F X([1 3])=exp(X([1 3])); X(2)=2*sinh(X(2)); Pars=[X]; 09/05/2019 10
Use Case Setup an EGI Notebook Using the RTG docker SW – Add a small set of data wget from eiscat portal – Insert analysis code First versions – Run RTG interactively Develop/Refjne code – Store analysis script(s) .m/.zip/.tgz – Run batched RTG process CC docker – Big data set – 09/05/2019 11
JupyterLab - Octave EGI Marketplace Order octave – Python/Julia/Octave/R T erminal → git, wget,... – 09/05/2019 12
JupyterLab - Octave Setup an EGI Notebook Using the RTG docker – Add a small set of data wget from eiscat portal – Insert analysis script(s) First versions – Run RTG interactively Refjne script(s) – Store analysis script(s) .m/.zip/.tgz – Run batched RTG process Big data set – 09/05/2019 13
Thank you Any questions?
Recommend
More recommend
