Grid Computing for I ndustry Grid Computing for I ndustry Early - - PowerPoint PPT Presentation

Grid Computing for I ndustry Grid Computing for I ndustry – – Early Applications Early Applications

Hing-Yan LEE National Grid Office Singapore

International Symposium on Grid Computing 2007

Preparing for I ndustry Adoption Preparing for I ndustry Adoption

Multi Multi-

• Organization Grid

Organization Grid Accounting System Accounting System (MOGAS) (MOGAS)

Resource Usage

…

Resource Usage Monitoring

Collection of raw data

Metering

Usage charging

Accounting

Organisational level consumer-provider business relationship

$ = f (CPU , memory, license,…) Bill for organization A = ∑ (usage of members

• f organization A at resources owned by other
• rganizations)

Ganglia

Only stores & reports on information of resource status, no information of users & their jobs Hence, no metering & accounting mechanism [Courtesy of A/Prof. Francis Lee, NTU]

PRAGMA MOGAS status PRAGMA MOGAS status (12/10/2006) (12/10/2006)

AIST, Japan CNIC, China KISTI, Korea ASCC, Taiwan NCHC, Taiwan UoHyd, India MU, Australia BII, Singapore KU, Thailand USM, Malaysia NCSA, USA SDSC, USA CICESE, Mexico UNAM, Mexico UChile, Chile TITECH, Japan

14 Organizations deployed with MOGAS, 5 of them with GT4 Cindy Zheng, GGF13, 12/10/06 modified by A/Prof. Bu-Sung Lee

MIMOS IOIT-HCM

GT4 GT2

NGO, Singapore QUT, Australia OSAKAU, Japan

Netrust Netrust Certificate Authority Certificate Authority

• Commercial CA & Singapore’s only Licensed CA
• Current Status:

– Completed migration. All existing NGPP sites have migrated their host certificates to Netrust certificates. – Continual effort to issue certificates for

• New users of NGPP resources
• Additional hosts added to NGPP

– Seeking accreditation from APGrid PMA (part of IGTF)

• Temporary CA will continue to exist to issue certificates

for testing & trials.

Meta Scheduler Rollout Meta Scheduler Rollout

• Objective

– To provide LSF Meta-Scheduler for seamless access to NGPP compute resources

• Collaboration with Platform Computing
• Meta-Scheduler inter-operates with

– LSF, PBS Pro & N1GE

• Completed: Q3 2005

NGPP Meta NGPP Meta-

• Scheduler

Scheduler

• The LSF Meta-Scheduler

interfaces with local workload schedulers

(e.g. Sun’s N1GE, Platform’s LSF) of resources on NGPP, and

schedules the job to the

best available resources.

• Current applications using
• r planning to use the

Meta-Scheduler include:

– MicroRNA project (BII) – Media Grid (AE@SG) – DMG Portal (NGPP) – Multipitch Speech project (I 2R)

NUS-SMA

Hydra3 60 CPUs

I MCB

4 CPUs

NGO-GOG

Soursop 78 CPUs

I HPC

Lime 6 CPUs

NGPP Solution NGPP Solution

Proof Proof-

• of
• f-
• Concepts

Concepts

Goal of Proof-of-Concept

For users to harness resources on NGPP to run FEMLAB on a pay-per-use basis resulting in cost-savings for companies, encouraging R&D activities, & improving utilization of resources.

Test Test-

• bedding of Provisioning

bedding of Provisioning

• Objective

– To assess viability of a spill-over facility using GOG clusters – To measure speed of provisioning – To explore appropriate cost model

• Results

– Accuracy of results confirmed – Mentor Graphics to review license conditions

Matlab Matlab POC POC

• Aim:

– To explore distributed version of MATLAB (Distributed Computing Toolbox) on GOG cluster

Digital Media Digital Media

Enterprise Grid for Enterprise Grid for SMEs SMEs

• Objectives

– Create awareness in SMEs – Provide assistance & guidance to SMEs to harness Grid Computing to exploit internal compute resources

• Beneficiaries

– MegaMedia’s digital media hub (dmh) – Ngee Ann Polytechnic – Singapore Polytechnic

Setting Up Setting Up – –

• Set up & configure a 10-node cluster with

mental ray software

• Able to successfully check out licenses from

NG’s license server to their compute cluster for

• rendering. Completed Jul 2006.
• Following-up with teachers of Digital Media

class for students to use the licenses for class assignments and projects.

• Same effort for

Rendering Grid Rendering Grid

• Objectives

– To provide licenses and machines to Digital Media companies for their rendering needs, as a means to move them to utilizing services on the Grid – Nurturing the emergence of a Grid Service Provider

• Mental ray licenses for 64 CPUs acquired

– Free access & no charge to commercial & tertiary users – Floating licenses – Support from mental images GmbH

• Compute resources from GOG machines

Use of mental ray Licenses Use of mental ray Licenses

• Digital Media companies that have used the facilities
• TheContentCompany
• Omens-Studio
• VHQ Productions
• Companies in discussion
• 3dSense (animation school – 3dsMax & Maya)
• Big Communications Pte Ltd (animation - Maya)
• ST Electronics (Digital Media) Pte Ltd formerly IV Labs

(animation & games)

• mega media (animation – Maya)
• Visual Mediaworks (animation – 3dsMax)
• Lights & Shadows (animation – 3dsMax)
• 7cans (animation & games – 3dsMax)
• 10tacle (games – 3dsMax)
• Frontline Technologies Pte Ltd (Grid Service Broker)

Usage Usage – – TheContentCompany TheContentCompany

• A 4-man company
• Title: The Yellowman Shortfilm
• Required resources for rendering a

submission to “Cannes Lions 2006” 53rd International Advertising Festival (18-24 June 2006)

Usage Usage – –

“Smoking Fish – Hold onto Your Butt”

• Public Service Announcement

commissioned by the Surfrider

Foundation of America to

educate the public about how

pollution is harming the country’s coastline.

• Worked in tandem with Hollywood writer Adam McKay (“Anchorman -

Legend of Ron Burgundy”, “Bewitched”) & director Ian O’Roarty

• Omens Studios produced the CG animation for the project. The PSA

was televised nationally in USA in 2006 on various networks.

Quotes Quotes

• “It is now easier and faster to visualize our

work, instead of waiting for days before the work is finished.”

Tang Chi Sim MD, Omens Studios

• “Grid Computing allows local digital media firms

to focus on the creative side of things, rather than worrying over the lack of computing resources.”

Yeo Chun Cheng CIO, Media Development Authority

Other Applications Other Applications

Land Data Hub Land Data Hub -

• 1

1

• A statutory board whose main focus in

land resource optimization

• Maintains the national repository of land

information (Land Data Hub or LDH)

– Collated from 14 public & private agencies – Shared across the public sector

• Project Objective

– Replace aging IT infrastructure with a more efficient & flexible architecture

Land Data Hub Land Data Hub -

• 2

2

• Motivated by

– Need to cut costs – Achieve efficient database & system management – Ensure high scalability & availability – Allow fast & convenient data sharing among LDH members

• Designed to be

– A platform for inter-agency collaboration to develop cross-agency data products & services for the public – An inter-agency consultation platform

Land Data Hub Land Data Hub -

• 3

3

• “Enterprise grid computing offers new

capabilities on how geospatial data can be shared at a potentially lower cost”

• “The Oracle 10g grid platform redefines

how data is collected and shared among agencies while delivering greater efficiencies at lower costs”

– Lewis Wu (Director, IT, SLA)

CrossWalk CrossWalk

• Users

– IT Department, National Library Board

• Challenge

– Reduce record conversion time from 3 full days to an 8-hour overnight run

• Domain

– Data processing for monthly reporting

• Solution

– Distribute the workload across as many idle computers as possible to convert the records in parallel using Condor workload management system – Instead of adding converted records t database residing on central server, each computer has its local database to store converted records – Removes bottleneck during records insertions into a central database, but requires an additional step of merging local databases subsequently

• Benefits

– Speedup of processing using idle compute resources within the enterprise, with no additional investment required.

CrossWalk CrossWalk Benchmark Benchmark -

• 2

2

• Input

– 50,000 records

• Output

– 50,000 records

• Benchmarking Result

– Original: 1 CPU takes about ~ 3 days – Grid: 4 CPUs take about ~ 3.5 hours

• Grid resources are not dedicated for the benchmark &

shared load with other jobs

– Will require about ~ 29 CPUs to achieve target processing time

• Users

– IT Department, National Library Board

• Challenge

– Undertake web archival with reasonable investment in compute resources

• Domain

– Crawl, index & archive web materials of specific interest to Singapore

• Solution

– Use available compute resources on NGPP to run NutchWAX indexing software to address scalability – Use distributed compute resources on NGPP to run Heritrix crawling software to partition crawling of web sites

• Benefits

– Benchmark results show:

• Frequency of index release can be maintained
• Scalability can be addressed up to a certain extent, depending on availability of

compute resources availability on NGPP

– Short-term: No need for purchase of additional computing resources

WAS WAS NutchWAX NutchWAX -

• 2

2

• Input

– 3,094 ARC files occupying 135 GB

• Output

– 72 GB

• Benchmarking Result

– Original: 1 CPU takes about ~ 72 hours – Grid: 72 CPUs take about ~ 14 hours

• Grid resources are not dedicated for the

benchmark & shared load with other jobs

Speech Modeling Speech Modeling

• Users

– Speech & Dialogue Processing Lab at Institute for Infocomm Research

• Challenge

– Process huge no. of voice data files (36 GB) using multi-pitch program

• Domain

– Identify & differentiate voices in sound recordings by identifying unique frequencies in each voice – Processing of input file & producing track files is inherently parallelizable

• Solution

– Re-structure source code of multi-pitch program – Use meta-scheduler to distribute tasks to available compute resources

• n NGPP
• Benefits

– Original time taken: 9 months to process on 1 CPU – Speedup: 2 days using 60 server class Xeons CPUs – Results validated by speech analysis expert

Speech Recordings Separate the pitch tracks (fundamental frequency) Determine if there are 0, 1 or 2 people speaking Detect single speaker segments

End End