Daiquiri an VO ready solution for medium size data providers Galkin - - PowerPoint PPT Presentation

daiquiri an vo ready solution
SMART_READER_LITE
LIVE PREVIEW

Daiquiri an VO ready solution for medium size data providers Galkin - - PowerPoint PPT Presentation

Daiquiri an VO ready solution for medium size data providers Galkin Anastasia Galkin Anastasia Klar Jochen Klar Jochen ic Gal Gal Matiev Matievic Harr Harry Enk y Enke e As Asterics data pro data provider f ider forum, 27.06.2018


slide-1
SLIDE 1

Anastasia Anastasia Galkin Galkin Jochen Jochen Klar Klar Gal Gal Matiev Matievic ic Harr Harry Enk y Enke e As Asterics data pro data provider f ider forum, 27.06.2018

  • rum, 27.06.2018

Daiquiri – an VO ready solution for medium size data providers

slide-2
SLIDE 2

Daiquiri

A A framew framework

  • rk fo

for the the publ publication ication of

  • f scientific

scientific databases databases

2

  • Allows for highly customizable web applications
  • Based on a common easily maintainable code base
  • Separated into an app and the daiquiri library
  • Features:
  • SQL web interface to relational databases
  • User space
  • User management and user registration work flow
  • Metadata management and access control
  • First glance in-browser plotting
  • Table download and file service
  • Employs VO protocols and standards
slide-3
SLIDE 3

Public Databases hosted @AIP

3

  • RAVE database query interface for DR3 (Siebert et al., 2011)
  • MultiDark Database (Riebe et al., 2011)
  • RAVE (2013), CosmoSim (2014), APPLAUSE (2015), Gaia@AIP (2016)
  • Future: 4MOST public archive
slide-4
SLIDE 4

4

slide-5
SLIDE 5

5

slide-6
SLIDE 6

6

slide-7
SLIDE 7

7

slide-8
SLIDE 8

Language and framework

8

Py Python thon and and Djang Django

  • Python: todays preferred scripting language, widely used in astronomy
  • Django: full MVC framework with everything included, huge community
  • Django REST framework: de-facto standard for REST interfaces in python
  • Django-allauth: local and social authentication, registration work flows
  • astropy: community python library for Astronomy
  • ...

Front-end

AngularJ AngularJS and and Bootstrap Bootstrap

  • AngularJS 1: awesome since Daiquiri v1
  • Bootstrap 3: responsive layout, mobile friendly
slide-9
SLIDE 9

Understanding queries

9

Quer Queryparser yparser based based on

  • n Antlr

Antlr

  • Antlr: parser generator for structured text or binary files
  • queryparser: Antlr generated python code to parse query strings
  • ADQL translator using ADQL grammar to translate to MySQL or PostgresQL

syntax

  • MySQL and PostgresQL parsers using MySQL / PostgresQL grammar for

parsing an SQL query

  • open source and available on GitHub and PyPI (python2, python3)
  • Using mysql_sphere to translate ADQL functions into MySQL
  • developed and maintained by Gal Matijevic (AIP)
slide-10
SLIDE 10

Asynchronous jobs

10

Ce Celery and RabbitMQ and old-school unix pipes

Downloading tables

Ce Celery and and RabbitMQ RabbitMQ

  • Celery: asynchronous task queue queue in Python, widely adopted
  • RabbitMQ: message broker in Erlang
  • redis: in-memory data structure store for task results, can also be used for
  • caching
  • systemd: new init system for Linux, make it easy to deploy daemons
slide-11
SLIDE 11

Features

11

Implemented Implemented

  • SQL query interface (with examples, job list, plotting, …)
  • Full ADQL + SQL syntax of PostgresQL – pgSphere integration
  • Customizable data query forms
  • Contact messages + ~mangement for the support staff
  • DOI integration and landing pages for databases and tables
  • File service and zip-archive creation
  • Registration and log-in using Oauth2 (facebook, twitter, GitHub, Google, ORCID)
  • WordPress as CMS for project presentation and documentation
  • VO protocols:
  • Data Access Layer Interface (DALI)
  • Table Access Protocol (TAP)
  • Universal Worker Service Pattern (UWS)
  • Co

Cone search

  • File access, filtering and download for observatory archives (MUSE WIDE)
  • Cut-out service for images and datacubes
slide-12
SLIDE 12

Features

12

Upc Upcoming

  • ming
  • FITS tables download
  • Use of sharded databases (paqu v2)
  • Management of project/collaboration meetings
  • VO protocols:
  • Simple Image Access (SIA)
  • Simple Spectral Access (SSA)
  • Provenance Data Model (ProvSAP, ProvTAP)
slide-13
SLIDE 13

Try django-daiquiri!

13

as as a a user user

Production version of the Gaia@AIP Services: https://gaia.aip.de/

as as a a pro provider ider

README: https://github.com/aipescience/django-daiquiri/blob/master/README.rst

slide-14
SLIDE 14

Daiquiri v2

14

djang django-daiquir

  • daiquiri (

i (sinc since 2016) e 2016) Python (>= 2.7 and >= 3.4) using the Django framework (1.11) Responsive front-end written in AngularJS 1 and Bootstrap 3

Using:

  • MariaDB 10.1 or PostgresQL 9.6/10
  • queryparser and Antlr to parse and translate queries
  • Celery, RabbitMQ, redis and systemd for asynchronous tasks (not only

queries)

  • Old school unix pipes to create files using mysqldump or pgdump
  • pen source and available on

GitHub and PyPy to deploy daemons

slide-15
SLIDE 15

Questions?

15

Anastasia Galkin agalkin@aip.de github.com/aipescience escience.aip.de