Daiquiri an VO ready solution for medium size data providers Galkin - - PowerPoint PPT Presentation

Anastasia Anastasia Galkin Galkin Jochen Jochen Klar Klar Gal Gal Matiev Matievic ic Harr Harry Enk y Enke e As Asterics data pro data provider f ider forum, 27.06.2018

• rum, 27.06.2018

Daiquiri – an VO ready solution for medium size data providers

Daiquiri

A A framew framework

• rk fo

for the the publ publication ication of

• f scientific

scientific databases databases

2

• Allows for highly customizable web applications
• Based on a common easily maintainable code base
• Separated into an app and the daiquiri library
• Features:
• SQL web interface to relational databases
• User space
• User management and user registration work flow
• Metadata management and access control
• First glance in-browser plotting
• Table download and file service
• Employs VO protocols and standards

Public Databases hosted @AIP

3

• RAVE database query interface for DR3 (Siebert et al., 2011)
• MultiDark Database (Riebe et al., 2011)
• RAVE (2013), CosmoSim (2014), APPLAUSE (2015), Gaia@AIP (2016)
• Future: 4MOST public archive

4

5

6

7

Language and framework

8

Py Python thon and and Djang Django

• Python: todays preferred scripting language, widely used in astronomy
• Django: full MVC framework with everything included, huge community
• Django REST framework: de-facto standard for REST interfaces in python
• Django-allauth: local and social authentication, registration work flows
• astropy: community python library for Astronomy
• ...

Front-end

AngularJ AngularJS and and Bootstrap Bootstrap

• AngularJS 1: awesome since Daiquiri v1
• Bootstrap 3: responsive layout, mobile friendly

Understanding queries

9

Quer Queryparser yparser based based on

• n Antlr

Antlr

• Antlr: parser generator for structured text or binary files
• queryparser: Antlr generated python code to parse query strings
• ADQL translator using ADQL grammar to translate to MySQL or PostgresQL

syntax

• MySQL and PostgresQL parsers using MySQL / PostgresQL grammar for

parsing an SQL query

• open source and available on GitHub and PyPI (python2, python3)
• Using mysql_sphere to translate ADQL functions into MySQL
• developed and maintained by Gal Matijevic (AIP)

Asynchronous jobs

10

Ce Celery and RabbitMQ and old-school unix pipes

Downloading tables

Ce Celery and and RabbitMQ RabbitMQ

• Celery: asynchronous task queue queue in Python, widely adopted
• RabbitMQ: message broker in Erlang
• redis: in-memory data structure store for task results, can also be used for
• caching
• systemd: new init system for Linux, make it easy to deploy daemons

Features

11

Implemented Implemented

• SQL query interface (with examples, job list, plotting, …)
• Full ADQL + SQL syntax of PostgresQL – pgSphere integration
• Customizable data query forms
• Contact messages + ~mangement for the support staff
• DOI integration and landing pages for databases and tables
• File service and zip-archive creation
• Registration and log-in using Oauth2 (facebook, twitter, GitHub, Google, ORCID)
• WordPress as CMS for project presentation and documentation
• VO protocols:
• Data Access Layer Interface (DALI)
• Table Access Protocol (TAP)
• Universal Worker Service Pattern (UWS)
• Co

Cone search

• File access, filtering and download for observatory archives (MUSE WIDE)
• Cut-out service for images and datacubes

Features

12

Upc Upcoming

• ming
• FITS tables download
• Use of sharded databases (paqu v2)
• Management of project/collaboration meetings
• VO protocols:
• Simple Image Access (SIA)
• Simple Spectral Access (SSA)
• Provenance Data Model (ProvSAP, ProvTAP)

Try django-daiquiri!

13

as as a a user user

Production version of the Gaia@AIP Services: https://gaia.aip.de/

as as a a pro provider ider

README: https://github.com/aipescience/django-daiquiri/blob/master/README.rst

Daiquiri v2

14

djang django-daiquir

• daiquiri (

i (sinc since 2016) e 2016) Python (>= 2.7 and >= 3.4) using the Django framework (1.11) Responsive front-end written in AngularJS 1 and Bootstrap 3

Using:

• MariaDB 10.1 or PostgresQL 9.6/10
• queryparser and Antlr to parse and translate queries
• Celery, RabbitMQ, redis and systemd for asynchronous tasks (not only

queries)

• Old school unix pipes to create files using mysqldump or pgdump
• pen source and available on

GitHub and PyPy to deploy daemons

Questions?

15

Anastasia Galkin agalkin@aip.de github.com/aipescience escience.aip.de