High Performance Streaming Data Processing F. Huizinga Radio - - PowerPoint PPT Presentation

▶
high performance streaming data processing
SMART_READER_LITE
LIVE PREVIEW

High Performance Streaming Data Processing F. Huizinga Radio - - PowerPoint PPT Presentation

Dec 29, 2022 15 likes •156 views

High Performance Streaming Data Processing F. Huizinga Radio Astronomy Study celestial objects in radio frequencies LOFAR ( LO w F requency AR ray) 7000 Antennas Below 250 MHz Multiple stations Biggest in the world

slide-1
SLIDE 1

High Performance Streaming Data Processing

  • F. Huizinga
slide-2
SLIDE 2

Radio Astronomy

  • Study celestial objects in radio frequencies
  • LOFAR (LOw Frequency ARray)

○ 7000 Antennas ○ Below 250 MHz ○ Multiple stations ○ Biggest in the world ○ New design

slide-3
SLIDE 3

Example image LOFAR

slide-4
SLIDE 4

LOFAR Superterp

  • 6 Stations
  • 288 usable ants
  • 300m diameter
slide-5
SLIDE 5

New Antenna Design

  • Direction by phase delays between antennas
  • Highly computational
  • Cheap antennas
slide-6
SLIDE 6

Enter The AARTFAAC Project

  • Amsterdam Astron Radio Transients Facility

And Analysis Centre

  • 120 degrees FOV
  • Realtime imaging
  • Looking for transients

○ Non static objects ○ Pulsars ○ Galaxies ○ Jets

  • Superterp only
  • Trigger LOFAR
slide-7
SLIDE 7

Computational Challenges

  • Correlator: 12 Uniboards of FPGAs @

Astron

  • Visibilities: (288*289)/2*4 = 166464 (dual

pol.)

  • Bandwidth: up to 13.8 MHz (576 images)
  • Dump time: Every 1/10th second
  • Data rate: 40 Gb/s
  • Commensal to LOFAR
  • Responsetime: <= 1 second
slide-8
SLIDE 8

Pipeline Framework Candidates

  • Hadoop

○ - No streaming data (batch processing)

  • Storm

○ + Twitter ○ + Define your own topology easily ○ - Meta data overhead

  • Pelican

○ + Close collaboration ○ + Astronomy examples/projects ○ - Young

  • 0-MQ

○ - Low level ○ + Large userbase, docs

slide-9
SLIDE 9

Pelican Framework (1/2)

  • Streaming data processing
  • Developed by OeRC (Oxford)
  • C/C++
  • Depends

○ Boost (minimal) ○ Qt framework ○ CppUnit ○ Doxygen ○ CMake

  • Designed with astronomy in mind
  • Modular
slide-10
SLIDE 10

Pelican Framework (2/2)

  • Emulator

○ Read from disk, self ○ Send udp packets

  • Server (bottleneck)

○ Creates chunks from udp ○ Sends tcp packets ○ First come first served

  • Pipeline

○ Creates datablobs from tcp ○ Manipulate datablobs with modules ○ Send/Store elsewhere

Emulator Pelican Server

Module 1 Module 2 Module N Module 1 Module 2 Module N

Pipelines

Module 1 Module 2 Module n

slide-11
SLIDE 11

Pipeline 2

Pipeline Design

Transient Pipeline Monitoring and Control Antenna Image cube Antenna Antenna Correlator Pipeline 1

Flagger Calibrator Imager Flagger Calibrator Imager

Pipeline n

Flagger Calibrator Imager

Server Calibrator

  • Matlab
  • Pelican
  • CppUnit
  • Eigen3
  • Jenkins
slide-12
SLIDE 12

Example Image AARTFAAC

slide-13
SLIDE 13

More info

  • Sourcecode

○ Sorry, not yet ○ Nearby future!! ○ Some legal issues

  • www.aartfaac.org
  • www.lofar.org
  • github.com/pelican/pelican
  • eigen.tuxfamily.org
slide-14
SLIDE 14

Questions

?