320454 Big Data Project A Instructor: Peter Baumann email: - - PowerPoint PPT Presentation

320302 Databases & Web Applications (P. Baumann)

320473 Databases & Web Applications Lab 320454 Big Data Project A

Instructor: Peter Baumann email: p.baumann@jacobs-university.de tel:

• 3178
• ffice:

room 88, Research 1

2 Databases & Web Applications Lab | Big Data Project (P. Baumann)

Big Science Data

[OGC Ocean Science Interoperability Experiment; image source: Mbari]

3 Databases & Web Applications Lab | Big Data Project (P. Baumann)

MultiSolid Coverage

OGC Coverage Types

• Coverage = digital representation of

space/time varying phenomenon

• n-D

«FeatureType»

Abstract Coverage MultiPoint Coverage MultiCurve Coverage MultiSurface Coverage Grid Coverage Referenceable GridCoverage Rectified GridCoverage

4 Databases & Web Applications Lab | Big Data Project (P. Baumann)

Facing the Coverage Deluge

coverage server

4

sensor feeds

[OGC SWE]

5 Databases & Web Applications Lab | Big Data Project (P. Baumann) [OGC SWE]

sensor feeds

Taming the Coverage Deluge

5

coverage server

6 Databases & Web Applications Lab | Big Data Project (P. Baumann)

Let’s Take a Closer Look...

• Divergent access patterns for ingest and retrieval
• Server must mediate between access patterns

t

7 Databases & Web Applications Lab | Big Data Project (P. Baumann)

Our Research

• Large-Scale Scientific Information Services (L-SIS) Research Group
• flexible, scalable services on massive multi-dimensional scientific data
• Particular focus: n-D arrays
• Massive = multi-TB … multi-PB per object
• Results:
• rasdaman array DBMS (www.rasdaman.org),

demo at www.earthlook.org

• Geoservice standards: OGC WCS suite,

http://external.opengeospatial.org/twiki_public/CoveragesDWG/WebHome

• ISO 9075 SQL Part 15: MDA (under work)

8 Databases & Web Applications Lab | Big Data Project (P. Baumann)

• „raster data manager“:

Array Database = SQL + n-D arrays

• “tile streaming” architecture:

scaling from laptop to cloud

rasdaman: Scalable Array Analytics

www.rasdaman.org rasdaman Web visitors

select img.green[x0:x1,y0:y1] > 130 from LandsatArchive as img

9 Databases & Web Applications Lab | Big Data Project (P. Baumann)

[Diedrich et al 2001]

Use Case: Satellite ImageTime Series

10 Databases & Web Applications Lab | Big Data Project (P. Baumann)

• Big Earth Data Analytics
• Up to 130 TB databases for all Earth sciences + planetary science
• EU FP7-INFRA, 3 years, 5.85 mEUR
• Platform: rasdaman; strictly open standards

Ea Eart rthServe Server

Planetary Science

Mars geology

Cryospheric Science

landcover mapping

Oceanography

marine model runs + in-situ data

Geology

geological models

Airborne Science

high-altitude drones

Atmospheric Science

climate variables

11 Databases & Web Applications Lab | Big Data Project (P. Baumann)

Database Visualization

select encode( struct { red: (char) s.image.b7[x0:x1,x0:x1], green: (char) s.image.b5[x0:x1,x0:x1], blue: (char) s.image.b0[x0:x1,x0:x1], alpha: (char) scale( d.elev, 20 ) }, "image/png" ) from SatImage as s, DEM as d

[JacobsU, Fraunhofer; data courtesy BGS, ESA]

[JacobsU, Fraunhofer 2012; [data courtesy BGS, ESA]

12 Databases & Web Applications Lab | Big Data Project (P. Baumann)

• ad-hoc federation
• mixed hardware

Parallel / Distributed Query Processing

Dataset B Dataset A Dataset D Dataset C select max((A.nir - A.red) / (A.nir + A.red))

• max((B.nir - B.red) / (B.nir + B.red))
• max((C.nir - C.red) / (C.nir + C.red))
• max((D.nir - D.red) / (D.nir + D.red))

from A, B, C, D

13 Databases & Web Applications Lab | Big Data Project (P. Baumann)

Secured Archive Integration

First-ever direct, ad-hoc mix from protected NASA & ESA services in OGC WCS/WCPS Web client (EarthServer + CobWeb)

14 Databases & Web Applications Lab | Big Data Project (P. Baumann)

Demo

15 Databases & Web Applications Lab | Big Data Project (P. Baumann)

Next: On-Board Query Intelligence

Democratize direct data access

[imagery courtesy ESA, NASA] [OPS-SAT: ESA CubeSat]

16 Databases & Web Applications Lab | Big Data Project (P. Baumann)

Summary

• Project work
• embedded in international projects & collaborations
• Present
• Publish

17 Databases & Web Applications Lab | Big Data Project (P. Baumann)

Big Picture

• 320302

Databases and Web Applications

• Fall lecture, undergrad + grad
• Advanced course in spring: Information Architectures
• 320473

Databases and Web Applications Lab

• Lab, grad
• 320454

Big Data Project A

• Project, grad

New meeting slot: Tue 09:45, Research 1, room 88

18 Databases & Web Applications Lab | Big Data Project (P. Baumann)

Project Task

• Pick a topic
• http://www.faculty.jacobs-university.de/pbaumann/iu-

bremen.de_pbaumann//Courses/ResearchTopics/

• Perform task – planful:
• Spec document

20%

• - Sep 26 Oct 03
• Prototype 1: breakthrough implementation

20%

• - Oct 17
• Prototype 2: ready for benchmark

20%

• - Oct 31
• Benchmark results

20%

• - Nov 14
• Publication

10%

• - Nov 28
• Prototype 3: ready for handover

10%

• - Dec 05

19 Databases & Web Applications Lab | Big Data Project (P. Baumann)

Resources

• rasdaman website
• www.rasdaman.org
• demo
• www.earthlook.org
• Our publications
• http://www.faculty.jacobs-university.de/pbaumann/iu-bremen.de_pbaumann//pubs.php
• Instructor:
• p.baumann@...
• ...and the rasdaman team

20 Databases & Web Applications Lab | Big Data Project (P. Baumann)

Main Evaluation Criteria

• complete wrt. requirements
• Solid engineering
• bug-free, project & code documentation, coding quality, ...
• user-friendliness and appealing look&feel
• complexity (in absolute terms and in comparison to other teams' work)
• Good writeup
• Specification, documentation, paper
• (no particular order)

21 Databases & Web Applications Lab | Big Data Project (P. Baumann)

Project/Lab Topics

• http://www.faculty.jacobs-university.de/pbaumann/iu-

bremen.de_pbaumann/teaching.php

• -> course list -> list of topics