an input output library for cluster of smp
play

An Input/Output LIbrary for cluster of SMP Adrien Lebre , Yves - PowerPoint PPT Presentation

Jan 15, 2024 •234 likes •629 views

An Input/Output LIbrary for cluster of SMP Adrien Lebre , Yves Denneulin { Adrien.Lebre,Yves.Denneulin } @imag.fr ID-IMAG (UMR 5132) Laboratory, Grenoble, France BULL - HPC, Echirolles, France. Slide 1/17 6th May 2005 aIOLi - CCGRID05 -

  1. An Input/Output LIbrary for cluster of SMP Adrien Lebre , Yves Denneulin { Adrien.Lebre,Yves.Denneulin } @imag.fr ID-IMAG (UMR 5132) Laboratory, Grenoble, France BULL - HPC, ´ Echirolles, France. Slide 1/17 6th May 2005 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  2. Plan 1 Introduction Context Parallel Input/Output 2 aIOLi system Preamble Principles Technical aspects 3 Results POSIX vs aIOLi MPI I/O vs aIOLi 4 Conclusion Slide 2/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  3. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Context Environment Cluster of SMPs Linux High Performance Computing Intensive I/O applications CPU bounded application ⇒ I/O bounded application Remote hard drive I/O Parallel I/O Handling concurrent accesses to a same resource (a file) Accesses : different in size, in offset Example : matrix product Slide 3/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  4. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Context Environment Cluster of SMPs Linux High Performance Computing Intensive I/O applications CPU bounded application ⇒ I/O bounded application Remote hard drive I/O Parallel I/O Handling concurrent accesses to a same resource (a file) Accesses : different in size, in offset Example : matrix product Slide 3/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  5. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Parallel I/O Example P0 read(fd,buf,1024); //file position=0 Matrix product Specific parts to fetch (according read(fd,buf,1024); //file position=1024 P1 to data distribution: columns, rows, BLOCK/BLOCK, read(fd,buf,1024); //file position=2048 P2 BLOCK/CYCLIC ...) Several requests at the same time : read(fd,buf,1024); //file position=3072 P3 disjoint/contiguous SMP Client “lethal” behavior for I/O subsystem 4 independent requests but contiguous from a global point of view Slide 4/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  6. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Parallel I/O Example Matrix product P0 read(fd,buf,1024); //file position=0 read(fd,buf,1024); //file position=4096 ... Specific parts to fetch (according read(fd,buf,1024); //file position=1024 P1 to data distribution: columns, read(fd,buf,1024); //file position=5120 rows, BLOCK/BLOCK, BLOCK/CYCLIC ...) read(fd,buf,1024); //file position=2048 P2 read(fd,buf,1024); //file position=6144 Several requests at the same time : read(fd,buf,1024); //file position=3072 P3 disjoint/contiguous read(fd,buf,1024); //file position=7168 “lethal” behavior for I/O subsystem SMP Client 4 requests have been processed ? What about the new requests ? Contiguous / Disjoint ? Slide 4/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  7. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Parallel I/O Example Matrix product P0 read(fd,buf,1024); //file position=0 read(fd,buf,1024); //file position=4096 ... Specific parts to fetch (according read(fd,buf,1024); //file position=1024 P1 to data distribution: columns, read(fd,buf,1024); //file position=5120 ... rows, BLOCK/BLOCK, BLOCK/CYCLIC ...) read(fd,buf,1024); //file position=2048 P2 read(fd,buf,1024); //file position=6144 ... Several requests at the same time : read(fd,buf,1024); //file position=3072 P3 disjoint/contiguous read(fd,buf,1024); //file position=7168 ... “lethal” behavior for I/O subsystem SMP Client 4 requests have been processed ? What about the new requests ? Contiguous / Disjoint ? Slide 4/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  8. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Parallel I/O Requirements / constraints Methods for disjoint data (readv) ⇒ complexity of API Collective operations ⇒ Synchronization mechanisms logical view (the files) ⇒ physical placements (block devices) Available solutions - related works Many Parallel File Systems : +/- efficient but hardware dependent “cluster compliant” : PVFS, NFSparallel, GPFS, Lustre Designed for “ Parallel I/O” : PIOUS, VESTA ... Libraries : Focus on portability aspects A lot ! : MPI I/O is the reference. Sophisticated API ⇒ Development overhead / Language bindings Slide 5/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  9. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Parallel I/O Requirements / constraints Methods for disjoint data (readv) ⇒ complexity of API Collective operations ⇒ Synchronization mechanisms logical view (the files) ⇒ physical placements (block devices) Available solutions - related works Many Parallel File Systems : +/- efficient but hardware dependent “cluster compliant” : PVFS, NFSparallel, GPFS, Lustre Designed for “ Parallel I/O” : PIOUS, VESTA ... Libraries : Focus on portability aspects A lot ! : MPI I/O is the reference. Sophisticated API ⇒ Development overhead / Language bindings Slide 5/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  10. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Context summary ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn IO server 1 IO server 2 IO server n Slide 6/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  11. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Context summary ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn IO server 1 IO server 2 IO server n Slide 6/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  12. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Context summary ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn IO server 1 IO server 2 IO server n Slide 6/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  13. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Context summary ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn IO server 1 IO server 2 IO server n Slide 6/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  14. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Context summary ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn IO server 1 IO server 2 IO server n Slide 6/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  15. Introduction Preamble aIOLi system Principles Results Technical aspects Conclusion aIOLi system Objectives Supply Parallel I/O algorithms scheduling policies aggregating access ⇒ efficiency overlapping access Only through the use of the ubiquitous POSIX calls open/creat/lseek/read/write/close ⇒ Simplicity Minimal overhead avoid expensive synchronisation mechanisms (barrier, . . . ) Slide 7/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  16. Introduction Preamble aIOLi system Principles Results Technical aspects Conclusion Evaluation of “the Linux” I/O stack 1 GB File decomposition on a SMP (kernel 2.4.27, IDPOT cluster, NFS version 3, mpich 1.2.5) 300 1 Observations 2 4 250 8 1 process ⇒ Sequential 1 randomize read (optimal) 200 Completion time + processes ⇒ - performance (sec) 150 1 process in random 100 access ⇒ more performance for large 50 access than parallel approach 0 1 4 8 16 32 64 128 512 1024 4096 Access granularity (KBytes) Slide 8/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Input and Output Objectives Survey IO facilities in .NET Framework Class Library file

Input and Output Objectives Survey IO facilities in .NET Framework Class Library file

Input and Output Objectives Survey IO facilities in .NET Framework Class Library file and directory management text files binary files object serialization 2 IO Library Input/output library in System.IO namespace

364 views • 32 slides
16. Recursion 2 Output: 103 Input: (3 + 5) * 20 Output: 160 Input: -(3 + 5) + 20 Output: 12

16. Recursion 2 Output: 103 Input: (3 + 5) * 20 Output: 160 Input: -(3 + 5) + 20 Output: 12

Motivation: Calculator Goal: we build a command line calculator Example Input: 3 + 5 Output: 8 Input: 3 / 5 Output: 0.6 Input: 3 + 5 * 20 16. Recursion 2 Output: 103 Input: (3 + 5) * 20 Output: 160 Input: -(3 + 5) + 20 Output: 12 Building a

387 views • 11 slides
17. Recursion 2 Input: 3 + 5 * 20 Output: 103 Input: (3 + 5) * 20 Output: 160 Input: -(3 + 5) + 20

17. Recursion 2 Input: 3 + 5 * 20 Output: 103 Input: (3 + 5) * 20 Output: 160 Input: -(3 + 5) + 20

Motivation: Calculator Goal: we build a command line calculator Example Input: 3 + 5 Output: 8 Input: 3 / 5 Output: 0.6 17. Recursion 2 Input: 3 + 5 * 20 Output: 103 Input: (3 + 5) * 20 Output: 160 Input: -(3 + 5) + 20 Building a Calculator,

245 views • 11 slides
7. Java Input/Output User Input/Console Output, File Input and Output (I/O) 133 User Input (half

7. Java Input/Output User Input/Console Output, File Input and Output (I/O) 133 User Input (half

7. Java Input/Output User Input/Console Output, File Input and Output (I/O) 133 User Input (half the truth) e.g. reading a number: int i = In.readInt(); Our class In provides various such methods. Some of those methods have to deal with wrong

1.09k views • 33 slides
BASIC INPUT/OUTPUT Fundamentals of Computer Science I Outline: Basic Input/Output Screen

BASIC INPUT/OUTPUT Fundamentals of Computer Science I Outline: Basic Input/Output Screen

BASIC INPUT/OUTPUT Fundamentals of Computer Science I Outline: Basic Input/Output Screen Output Keyboard Input Command Line Input File Input Simple Screen Output print("The count is " + str(count)) Outputs the

523 views • 27 slides
The Stream Hierarchy Inheritance of istream and ostream from ios ios istream ostream Stream

The Stream Hierarchy Inheritance of istream and ostream from ios ios istream ostream Stream

17 . Input/Output Mechanism: Standard IO, File IO Standard Input/Output stdin cin stdout cout stderr cerr Input/Output Streams: istream & ostream Input/Output of Built-in Types Input/Output of User-defined Types The Stream Hierarchy

356 views • 4 slides
Problems A problem (in Computer Science) specifies an input/output relationship, P I O . A

Problems A problem (in Computer Science) specifies an input/output relationship, P I O . A

1/23/2010 Problems A problem (in Computer Science) specifies an input/output relationship, P I O . A Constant-Space Model of Computation for First-Order Queries (input) (output) P Steven Lindell How are input/output represented as

412 views • 5 slides
some channel models Input X P(y|x) output Y transition probabilities memoryless: - output at

some channel models Input X P(y|x) output Y transition probabilities memoryless: - output at

some channel models Input X P(y|x) output Y transition probabilities memoryless: - output at time i depends only on input at time i - input and output alphabet finite Example: binary symmetric channel (BSC) 1-p Error Source 0 0 E p Y

528 views • 23 slides
BASIC INPUT/OUTPUT Fundamentals of Computer Science Outline: Basic Input/Output Screen Output

BASIC INPUT/OUTPUT Fundamentals of Computer Science Outline: Basic Input/Output Screen Output

BASIC INPUT/OUTPUT Fundamentals of Computer Science Outline: Basic Input/Output Screen Output Keyboard Input Simple Screen Output System.out.println("The count is " + count); Outputs the sting literal "the count is

651 views • 23 slides
18. Java Input/Output e.g. reading a number: int i = In.readInt(); Our class In provides various

18. Java Input/Output e.g. reading a number: int i = In.readInt(); Our class In provides various

User Input (half the truth) 18. Java Input/Output e.g. reading a number: int i = In.readInt(); Our class In provides various such methods. Some of those methods have to deal with wrong inputs: User Input/Console Output, File Input and Output

114 views • 9 slides
Java Input/Output 6 May 2019 OSU CSE 1 Overview The Java I/O (Input/Output) package java.io

Java Input/Output 6 May 2019 OSU CSE 1 Overview The Java I/O (Input/Output) package java.io

Java Input/Output 6 May 2019 OSU CSE 1 Overview The Java I/O (Input/Output) package java.io contains a group of interfaces and classes similar to the OSU CSE components SimpleReader and SimpleWriter component families Except that

855 views • 69 slides
An example grammar Todays Agenda: Input-Output Issues s --> np, vp. np --> [john]. vp

An example grammar Todays Agenda: Input-Output Issues s --> np, vp. np --> [john]. vp

Some hints regarding input/output with your projects Macros in Prolog Leftovers: The mystery of terminals on the LHS in DCGs A useful extension of print/1 : format/2 Changing the output behavior of the top-level A hook into

504 views • 5 slides
File Input and Output File Input and Output 1 / 9 File input/output input function reads values

File Input and Output File Input and Output 1 / 9 File input/output input function reads values

File Input and Output File Input and Output 1 / 9 File input/output input function reads values entered by a user. File Input and Output 2 / 9 File input/output input function reads values entered by a user. load function read data from a

450 views • 23 slides
Input/Output Cmd Line Input Formatted I/O Formatted Output Formatted Input Volker Sorge

Input/Output Cmd Line Input Formatted I/O Formatted Output Formatted Input Volker Sorge

SSC1: I/O Volker Sorge Overview I/O Streams Basic Streams Data Streams Software Systems Components 1 Standard Streams I/O Redirection Input/Output Cmd Line Input Formatted I/O Formatted Output Formatted Input Volker Sorge Buffered

1.14k views • 59 slides
lecture 19 Input / Output (I/O) - system bus - input e.g. keyboard, mouse - output

lecture 19 Input / Output (I/O) - system bus - input e.g. keyboard, mouse - output

lecture 19 Input / Output (I/O) - system bus - input e.g. keyboard, mouse - output e.g. printer, display screen Mon. March 21, 2016 System "Bus" (up to mid 1990's only) system bus It is cheaper and simpler to have

426 views • 23 slides
Multiple Input and Output Channels Multiple Input and Output Channels Multiple Input Channels In

Multiple Input and Output Channels Multiple Input and Output Channels Multiple Input Channels In

2/23/2019 channels slides Multiple Input and Output Channels Multiple Input and Output Channels Multiple Input Channels In [1]: import d2l from mxnet import nd def corr2d_multi_in(X, K): # First, traverse along the 0th dimension (channel

457 views • 6 slides
ODOT 5310 Program Development Updates Current Efforts & Next Steps TODAYs AGENDA

ODOT 5310 Program Development Updates Current Efforts & Next Steps TODAYs AGENDA

ODOT 5310 Program Development Updates Current Efforts & Next Steps TODAYs AGENDA Introduction of ODOT & Consultant Teams ODOT Plan Hierarchy Oklahoma Mobility Recap FTA 5310 Program and Next Steps Questions &

671 views • 44 slides
Timeline for AQ Activities 1 st Jan - SMP AQ Calculation (rolling AQ) * 1 st Dec - Snapshot for

Timeline for AQ Activities 1 st Jan - SMP AQ Calculation (rolling AQ) * 1 st Dec - Snapshot for

Timeline for AQ Activities 1 st Jan - SMP AQ Calculation (rolling AQ) * 1 st Dec - Snapshot for Formula Year AQ Value * 1 st Feb - SMP AQ Calculation December J a n (rolling AQ) u a r y * Supply Meter Point AQ will calculate if a

277 views • 4 slides
CSO Program Stakeholder Workgroup: Meeting #8 System Master Plan Control Options City Hall

CSO Program Stakeholder Workgroup: Meeting #8 System Master Plan Control Options City Hall

CSO Program Stakeholder Workgroup: Meeting #8 System Master Plan Control Options City Hall Council Chambers September 6, 2012 1 Welcome & Introductions City Representatives Julia Forgue Director of Utilities CH2M HILL

429 views • 25 slides
White Rose Doctoral Training Partnership Located at: University of Sheffield, Interdisciplinary

White Rose Doctoral Training Partnership Located at: University of Sheffield, Interdisciplinary

Best Student Presentation Form Pathway: SMP Students Name: Wanwan Zhu University: Leeds CEO Succession in Large U.S. Listed Presentation Title: Banks: Do Successor Characteristics Matter? Reason why selected as Good presentation to an

183 views • 6 slides
Impact of State Variation in Scope of Practice on Psychiatric Mental Health Nurse Practitioner

Impact of State Variation in Scope of Practice on Psychiatric Mental Health Nurse Practitioner

Impact of State Variation in Scope of Practice on Psychiatric Mental Health Nurse Practitioner Services AcademyHealth June 2, 2019 Susan A. Chapman Christopher Toretsky Bethany Phoenix Disclosures No financial or other conflicts of interest

375 views • 18 slides
RESEARCH PRESENTATION AND REPORT Prepared by Dr Joanne Chan (2018) 1 COVER PAGE (Topic)

RESEARCH PRESENTATION AND REPORT Prepared by Dr Joanne Chan (2018) 1 COVER PAGE (Topic)

RESEARCH PRESENTATION AND REPORT Prepared by Dr Joanne Chan (2018) 1 COVER PAGE (Topic) (Tutorial Section) (Group number) (Student names , Student numbers) 2 1. INTRODUCTION 1.1 Literature Review Definition of the key concepts e.g.

411 views • 14 slides
P ti Patient Centered Medical Care: t C t d M di l C Vision to Reality Implementing Care

P ti Patient Centered Medical Care: t C t d M di l C Vision to Reality Implementing Care

P ti Patient Centered Medical Care: t C t d M di l C Vision to Reality Implementing Care Management for Complex Management for Complex Patients in Primary Care Clemens Hong MD, MPH Grantmakers in Health 2012 Fall Forum Health Care

369 views • 21 slides
HEALTHY HOMES Masters in Public Policy INITIATIVE Practicum Prepared for: New Jersey Housing

HEALTHY HOMES Masters in Public Policy INITIATIVE Practicum Prepared for: New Jersey Housing

Edward J. Bloustein School of Planning and Public Policy HEALTHY HOMES Masters in Public Policy INITIATIVE Practicum Prepared for: New Jersey Housing and Mortgage Finance Agency Prepared by: Matthew Camarda, MPP Laraib Kamal, MPP-MPI Kurt

517 views • 34 slides

More recommend