WISENT Distributed processing of energy meteorologic data with - - PowerPoint PPT Presentation

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 1

WISENT

Distributed processing of energy meteorologic data with Condor

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 2

Outline

Project data & goals Overview of applications Introduction to Condor Live application demo

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 3

Project Data

BMBF funding programme: e-Science and Networked Knowledge Management Funding agency: DLR NMB+F Project duration: 10/2005 till 09/2008 Awarded project budget: ca. 2 Million EUR 7½ man-years over 3 calendar years Integrated into German D-Grid initiative Participants:

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 4

Energy Meteorology

An objective of energy meteorology is to obtain the information needed to characterize the fluctuating generation of solar and wind energy. Knowledge acquisition based on interdisciplinary cooperation: physical and meteorological methods transformation of wind and solar energy (physics) energy supply structures (electrical engineering, economics) monitoring methods (computer science, sensor technology) efficient, flexible distributed systems (computer science) Challenges of energy meteorology: preservation of future energy supply large amounts of data (terabytes) complex process chains (partially in real-time)

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 5

WISENT Objectives

WISENT supports the scientific collaboration of the energy meteorology community by enabling access to Grid technology. WISENT aims to extend and renew today's processes to support new scientific methods and business segments.

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 6

Work Package Overview

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 7

Work Packages 1-3

WP1: Use of Grid technologies in Virtual Organizations

• building a Grid infrastructure for scientific collaboration
• tools for networked knowledge and information management
• Virtual Institute of Energy Meteorology (vIEM) as an institutional framework

WP2: Transfer and integration of data

• automation, standardization and monitoring of data transfers
• foundation for distributed computing

WP3: Domain-specific standards and semantic interoperability

• standardization of interfaces
• evaluation of data quality
• interoperability between software systems of the project partners

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 8

Work Packages 4-7

WP4: Efficient interaction with large data objects

• visualization of large amounts of data
• new forms of visualization
• preemptive data distribution in the Grid

WP5: Distributed processing of energy meteorologic data

• Grid Computing: parallelizing of computations
• scalability advantages for existing systems
• enabling new application fields through Grid technology
• flexible use of external Grid resources

WP6: Cooperation with other Grid communities

• D-Grid; establishment of the „Energy Meteorology“ community
• EU project EGEE (Enabling Grids for E-sciencE)

WP7: Project coordination

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 9

WP1: TikiWiki Collaboration System

http://wisent.d-grid.de

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 10

WP2: Data Transfer meteocontrol <-> Uni OL

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 11

Photovoltaic Module Monitoring by meteocontrol

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 12

STEPS – Finding Locations for Solar Power Plants

Source: DLR-TT

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 13

3D Simulation of Solar Radiation Transfer

Source: DLR-IPA

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 14

3D Simulation of Solar Radiation Transfer, cont.

Source: DLR-IPA

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 15

Cloud Index Computation

Raw data, Source: Uni OL

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 16

Cloud Index Computation, cont.

Ground albedo, Source: Uni OL

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 17

Cloud Index Computation, cont.

Output, Source: Uni OL

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 18

Bottom-Up Introduction of Grid Middleware

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 19

Condor

Workload management system for compute-intensive jobs Developed at University of Winsconsin-Madison In production for >15 years Deployed world-wide in >1600 pools managing >100,000 hosts Keywords: high-throughput computing (not HPC!)

• pportunistic scheduling

cycle scavenging desktop Grids

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 20

Condor Features

Condor balances the conflicting requirements of resource users resource owners Condor manages heterogenous computing resources different types: from workstations to dedicated clusters different platforms (Unix, Windows) Similar middleware products: Portable Batch System (PBS) Load Sharing Facility (LSF) Sun Grid Engine IBM LoadLeveler

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 21

Condor Features, cont.

Job queuing Flexible scheduling based on user-defined priorities Resource monitoring Resource management Basic use scenario: Users submit one or more compute jobs to a Condor pool Condor decides when and where to execute jobs Condor monitors progress of each job's execution Users are notified upon their jobs' completion

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 22

Basic Architecture of a Condor Pool

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 23

Condor Jobs

A job consists of a (batch) executable with associated input/output files and a description of the desired execution context. (Optional) Stage-in/stage-out for files Restricted communication among jobs no support for interactive jobs distributed memory model independent job execution (possibly with input/output dependencies) limited support for message passing ideal for data parallelization, little use for program parallelization emphasis on throughput, not execution time

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 24

Condor Jobs, cont.

Condor universes (job categories) standard vanilla Java parallel ... The target universe for a job... determines the runtime environment in which the job is executed (e.g., sandboxing) imposes restrictions on what the executable may (not) do defines a specific set of features (e.g., process migration & checkpointing)

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 25

ClassAds

Schedulers advertise job requirements from their queue. Resource owners advertise the type and availability of resources. Central mgr matches both types of ads and arranges job executions. Source: Thain et. al, 2004

Jan Ploski Business Information Management OFFIS September 14th, 2006 Slide 26

Condor and the Grid

Run behind a higher-level Grid middleware (e.g., Globus, UNICORE) Use native support for inter-pool communication - „flocking“ Use the grid universe to submit Condor jobs to remote queues Interoperates with: Condor, Globus Toolkit, NorduGrid, Unicore, LSF, PBS Use the glidein mechanism for adding remote resources to the pool