WISENT Distributed processing of energy meteorologic data with Condor Jan Ploski September 14th, 2006 Slide 1 Business Information Management OFFIS
Outline Project data & goals Overview of applications Introduction to Condor Live application demo Jan Ploski September 14th, 2006 Slide 2 Business Information Management OFFIS
Project Data BMBF funding programme: e-Science and Networked Knowledge Management Funding agency: DLR NMB+F Project duration: 10/2005 till 09/2008 Awarded project budget: ca. 2 Million EUR 7½ man-years over 3 calendar years Integrated into German D-Grid initiative Participants: Jan Ploski September 14th, 2006 Slide 3 Business Information Management OFFIS
Energy Meteorology An objective of energy meteorology is to obtain the information needed to characterize the fluctuating generation of solar and wind energy. Knowledge acquisition based on interdisciplinary cooperation: physical and meteorological methods transformation of wind and solar energy (physics) energy supply structures (electrical engineering, economics) monitoring methods (computer science, sensor technology) efficient, flexible distributed systems (computer science) Challenges of energy meteorology: preservation of future energy supply large amounts of data (terabytes) complex process chains (partially in real-time) Jan Ploski September 14th, 2006 Slide 4 Business Information Management OFFIS
WISENT Objectives WISENT supports the scientific collaboration of the energy meteorology community by enabling access to Grid technology. WISENT aims to extend and renew today's processes to support new scientific methods and business segments. Jan Ploski September 14th, 2006 Slide 5 Business Information Management OFFIS
Work Package Overview Jan Ploski September 14th, 2006 Slide 6 Business Information Management OFFIS
Work Packages 1-3 WP1: Use of Grid technologies in Virtual Organizations • building a Grid infrastructure for scientific collaboration • tools for networked knowledge and information management • Virtual Institute of Energy Meteorology (vIEM) as an institutional framework WP2: Transfer and integration of data • automation, standardization and monitoring of data transfers • foundation for distributed computing WP3: Domain-specific standards and semantic interoperability • standardization of interfaces • evaluation of data quality • interoperability between software systems of the project partners Jan Ploski September 14th, 2006 Slide 7 Business Information Management OFFIS
Work Packages 4-7 WP4: Efficient interaction with large data objects • visualization of large amounts of data • new forms of visualization • preemptive data distribution in the Grid WP5: Distributed processing of energy meteorologic data • Grid Computing: parallelizing of computations • scalability advantages for existing systems • enabling new application fields through Grid technology • flexible use of external Grid resources WP6: Cooperation with other Grid communities • D-Grid; establishment of the „Energy Meteorology“ community • EU project EGEE (Enabling Grids for E-sciencE) WP7: Project coordination Jan Ploski September 14th, 2006 Slide 8 Business Information Management OFFIS
WP1: TikiWiki Collaboration System http://wisent.d-grid.de Jan Ploski September 14th, 2006 Slide 9 Business Information Management OFFIS
WP2: Data Transfer meteocontrol <-> Uni OL Jan Ploski September 14th, 2006 Slide 10 Business Information Management OFFIS
Photovoltaic Module Monitoring by meteocontrol Jan Ploski September 14th, 2006 Slide 11 Business Information Management OFFIS
STEPS – Finding Locations for Solar Power Plants Source: DLR-TT Jan Ploski September 14th, 2006 Slide 12 Business Information Management OFFIS
3D Simulation of Solar Radiation Transfer Source: DLR-IPA Jan Ploski September 14th, 2006 Slide 13 Business Information Management OFFIS
3D Simulation of Solar Radiation Transfer, cont. Source: DLR-IPA Jan Ploski September 14th, 2006 Slide 14 Business Information Management OFFIS
Cloud Index Computation Raw data, Source: Uni OL Jan Ploski September 14th, 2006 Slide 15 Business Information Management OFFIS
Cloud Index Computation, cont. Ground albedo, Source: Uni OL Jan Ploski September 14th, 2006 Slide 16 Business Information Management OFFIS
Cloud Index Computation, cont. Output, Source: Uni OL Jan Ploski September 14th, 2006 Slide 17 Business Information Management OFFIS
Bottom-Up Introduction of Grid Middleware Jan Ploski September 14th, 2006 Slide 18 Business Information Management OFFIS
Condor Workload management system for compute-intensive jobs Developed at University of Winsconsin-Madison In production for >15 years Deployed world-wide in >1600 pools managing >100,000 hosts Keywords: high-throughput computing (not HPC!) opportunistic scheduling cycle scavenging desktop Grids Jan Ploski September 14th, 2006 Slide 19 Business Information Management OFFIS
Condor Features Condor balances the conflicting requirements of resource users resource owners Condor manages heterogenous computing resources different types: from workstations to dedicated clusters different platforms (Unix, Windows) Similar middleware products: Portable Batch System (PBS) Load Sharing Facility (LSF) Sun Grid Engine IBM LoadLeveler Jan Ploski September 14th, 2006 Slide 20 Business Information Management OFFIS
Condor Features, cont. Job queuing Flexible scheduling based on user-defined priorities Resource monitoring Resource management Basic use scenario: Users submit one or more compute jobs to a Condor pool Condor decides when and where to execute jobs Condor monitors progress of each job's execution Users are notified upon their jobs' completion Jan Ploski September 14th, 2006 Slide 21 Business Information Management OFFIS
Basic Architecture of a Condor Pool Jan Ploski September 14th, 2006 Slide 22 Business Information Management OFFIS
Condor Jobs A job consists of a (batch) executable with associated input/output files and a description of the desired execution context. (Optional) Stage-in/stage-out for files Restricted communication among jobs no support for interactive jobs distributed memory model independent job execution (possibly with input/output dependencies) limited support for message passing ideal for data parallelization, little use for program parallelization emphasis on throughput, not execution time Jan Ploski September 14th, 2006 Slide 23 Business Information Management OFFIS
Condor Jobs, cont. Condor universes (job categories) standard vanilla Java parallel ... The target universe for a job... determines the runtime environment in which the job is executed (e.g., sandboxing) imposes restrictions on what the executable may (not) do defines a specific set of features (e.g., process migration & checkpointing) Jan Ploski September 14th, 2006 Slide 24 Business Information Management OFFIS
ClassAds Schedulers advertise job requirements from their queue. Resource owners advertise the type and availability of resources. Central mgr matches both types of ads and arranges job executions. Source: Thain et. al, 2004 Jan Ploski September 14th, 2006 Slide 25 Business Information Management OFFIS
Condor and the Grid Run behind a higher-level Grid middleware (e.g., Globus, UNICORE) Use native support for inter-pool communication - „flocking“ Use the grid universe to submit Condor jobs to remote queues Interoperates with: Condor, Globus Toolkit, NorduGrid, Unicore, LSF, PBS Use the glidein mechanism for adding remote resources to the pool Jan Ploski September 14th, 2006 Slide 26 Business Information Management OFFIS
Recommend
More recommend
