Lecture 21: Parallel Filesystems Abhinav Bhatele, Department of - PowerPoint PPT Presentation

Apr 06, 2023 •335 likes •455 views

Introduction to Parallel Computing (CMSC498X / CMSC818X) Lecture 21: Parallel Filesystems Abhinav Bhatele, Department of Computer Science Announcements Interim report was due yesterday Assignment 4 is due next week Abhinav Bhatele

Introduction to Parallel Computing (CMSC498X / CMSC818X) Lecture 21: Parallel Filesystems Abhinav Bhatele, Department of Computer Science
Announcements • Interim report was due yesterday • Assignment 4 is due next week Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING 2
Parallel filesystem • Home directories and scratch space typically on a parallel file system • Mounted on all login and compute nodes • Also referred to as I/O sub-system http://wiki.lustre.org/Introduction_to_Lustre Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING 3
Links between cluster and filesystem 36 total 9 9 9 9 9 9 Leaf Switch Each SU (1 management node, 1 login node, 2 LNET router nodes, 2 gateway nodes) Compute node LNET router node Object storage server (OSS) 9 9 Leaf Switch 9 total Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING 4
Different parallel filesystems • Lustre: open-source (lustre.org) • GPFS: General Parallel File System from IBM, now called Spectrum Scale • PVFS: Parallel Virtual File System Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING 5
Tape drive • Store data on magnetic tapes • Used for archiving data • Use robotic arms to access the right tape: https://www.youtube.com/watch?v=d- eWDuEo-3Q Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING 6
Burst buffer • Fast, intermediate storage between compute nodes and the parallel filesystem • Two designs: • Node-local burst buffer • Remote (shared) burst buffer Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING 7
I/O libraries • High-level libraries: HDF5, NetCDF • Middleware: MPI-IO • Low-level: POSIX IO Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING 8
Different I/O patterns • One process reading/writing all the data • Multiple processes reading/writing data from/to shared file • Multiple processes reading/writing data from/to different files • Different performance depending upon number of readers/writers, file sizes, filesystem etc. Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING 9
I/O profiling tools • Darshan • Lightweight profiling tool from Argonne National Lab • Recorder • Research prototype from UIUC Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING 10
Abhinav Bhatele 5218 Brendan Iribe Center (IRB) / College Park, MD 20742 phone: 301.405.4507 / e-mail: bhatele@cs.umd.edu

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Data Management Parallel Filesystems Dr David Henty HPC Training and Support

Data Management Parallel Filesystems Dr David Henty HPC Training and Support d.henty@epcc.ed.ac.uk +44 131 650 5960 Overview Lecture will cover Why is IO difficult Why is parallel IO even worse Lustre GPFS Performance

496 views • 25 slides

Taming Metadata Storms in Parallel Filesystems with MetaFS Tim Shaffer Motivation A

Taming Metadata Storms in Parallel Filesystems with MetaFS Tim Shaffer Motivation A (well-meaning) user tried to run a bioinformatics pipeline to analyze a batch of genomic data. MAKER 2 Motivation Shared filesystem performance became

507 views • 35 slides

Introduction to serial HDF5 Matthieu Haefele Saclay, April 2018, Parallel filesystems and

Introduction to serial HDF5 Matthieu Haefele Saclay, April 2018, Parallel filesystems and parallel IO libraries PATC@MdS Matthieu Haefele Training outline Day 1: AM: Serial HDF5 (M. Haefele) PM: Parallel IO and parallel HDF5 (M. Haefele)

1.09k views • 47 slides

Applying Machine Learning to Understand Write Performance of Large-scale Parallel Filesystems

Applying Machine Learning to Understand Write Performance of Large-scale Parallel Filesystems Presented by Bing Xie Bing Xie, Zilong Tan, Philip Carns, Jeff Chase, Kevin Harms, Jay Lofstead, Sarp Oral, Sudharshan S. Vazhkudai, Feiyi Wang ORNL

222 views • 18 slides

Mixing Hadoop and HPC Workloads on Parallel Filesystems Esteban Molina-Estolano * , Maya Gokhale

Mixing Hadoop and HPC Workloads on Parallel Filesystems Esteban Molina-Estolano * , Maya Gokhale , Carlos Maltzahn * , John May , John Bent , Scott Brandt * * UC Santa Cruz, ISSDM, PDSI Lawrence Livermore National Laboratory

516 views • 22 slides

Parallel Data Migration Between GPFS Filesystems via the iRODS Rule Engine Ilari Korhonen, PDC

Parallel Data Migration Between GPFS Filesystems via the iRODS Rule Engine Ilari Korhonen, PDC Center for High Performance Computing The 12th Annual iRODS Users Group Meeting June 9th 2020, The Internet Background PDC is a HPC center based

236 views • 19 slides

I/O 2 / Filesystems 1 1 Changelog Changes made in this version not seen in fjrst lecture: 13

I/O 2 / Filesystems 1 1 Changelog Changes made in this version not seen in fjrst lecture: 13 November: Correct cluster number on FAT directory entry slide. 1 last time page replacement modifjcations for scanning Linux: guess fjle pages used

2.03k views • 95 slides

This time we'll talk about filesystems. We'll start out by looking at disk partitions, which are

Linux System Administration SSU: Disks and Filesystems 1 This time we'll talk about filesystems. We'll start out by looking at disk partitions, which are the traditional places to put filesystems. Then we'll take a look at logical

711 views • 58 slides

Introduction Introduction to storage and to storage and filesystems filesystems Introduction

Moreno Baricevic Gilberto Daz Axel Kohlmeyer Stefano Cozzini ULA ICTP CNR-IOM DEMOCRITOS Merida, VENEZUELA Trieste, ITALY Trieste, ITALY Introduction Introduction to storage and to storage and filesystems filesystems Introduction

1.19k views • 51 slides

Parallel IO concepts (MPI-IO and pHDF5) Matthieu Haefele Saclay, April 2018, Parallel

Parallel IO concepts (MPI-IO and pHDF5) Matthieu Haefele Saclay, April 2018, Parallel filesystems and parallel IO libraries PATC@MdS Matthieu Haefele Outline Day 1 Morning: HDF5 in the context of Input/Output (IO) HDF5 Application

431 views • 32 slides

1 Analysis of sequential algorithms: The PRAM Model a Parallel RAM RAM model (Random Access

Outline Lecture 1: Multicore Architecture Concepts Lecture 2: Parallel programming with threads and tasks Design and Analysis Lecture 3: Shared memory architecture concepts of Parallel Programs Lecture 4: Design and analysis of parallel

356 views • 8 slides

Lecture 10: Parallel Patterns: The What and How of Parallel Programming G63.2011.002/G22.2945.001

Lecture 10: Parallel Patterns: The What and How of Parallel Programming G63.2011.002/G22.2945.001 November 9, 2010 Embarrassing Partition Pipelines Reduction Scan Tentative Plan for Rest of Class Today: Parallel Patterns Nov 16:

989 views • 86 slides

Lecture 2: Parallel Architectures Lecture 2: Parallel Architectures and Programming Models

ECE 451/566 - Intro. to Parallel & Distributed Prog. ECE-451/ECE-566 - Introduction to Parallel and Distributed Programming Lecture 2: Parallel Architectures Lecture 2: Parallel Architectures and Programming Models Department of Electrical

480 views • 32 slides

Shared Memory Programming with OpenMP Lecture 3: Parallel Regions Parallel region directive

Shared Memory Programming with OpenMP Lecture 3: Parallel Regions Parallel region directive Code within a parallel region is executed by all threads. Syntax: Fortran: !$OMP PARALLEL block !$OMP END PARALLEL C/C++: #pragma omp

321 views • 15 slides

STORAGE@TGCC & LUSTRE FILESYSTEMS WORKING & BEST PRACTICES Philippe DENIEL | |

STORAGE@TGCC & LUSTRE FILESYSTEMS WORKING & BEST PRACTICES Philippe DENIEL | | CEA/DAM/DIF PATC PARALLEL I/O APRIL 25TH, 2018 23 AVRIL 2018 CEA | 10 AVRIL 2012 | PAGE 1 AGENDA TGCC storage architecture TGCC storage workspaces

484 views • 26 slides

An Introduction to the Lustre Parallel File System Tom Edwards tedwards@cray.com C O M P U T E

An Introduction to the Lustre Parallel File System Tom Edwards tedwards@cray.com C O M P U T E | S T O R E | A N A L Y Z E Agenda Introduction to storage hardware RAID Parallel Filesystems Lustre Mapping

565 views • 35 slides

Transferring Data Between Applications Handling the system clipboard Drag-and-Drop operations

Transferring Data Between Applications Handling the system clipboard Drag-and-Drop operations Data transfer classes Data Transfer GUI architectures require a mechanism to Drag-drop is move data between application windows. expected with

416 views • 9 slides

Getting Started with Software Development for Cisco DevNet LEARNING THE FOUNDATIONS OF SOFTWARE

Getting Started with Software Development for Cisco DevNet LEARNING THE FOUNDATIONS OF SOFTWARE DESIGN Nick Russo NETWORK ENGINEER @nickrusso42518 www.njrusmc.net Should I be here? Agenda Core skills: Bash Basic software design skills

408 views • 22 slides

Spiritual Maturity: Be Filled with the Spirit Bruce Baker, Pastor Washington County Bible

Spiritual Maturity: Be Filled with the Spirit Bruce Baker, Pastor Washington County Bible Church Definition of Spiritual Maturity The spiritually mature are those who, through the power of the Holy Spirit , (1) understand Gods Word well

532 views • 29 slides

Users' consent - simple as SAML David Simonsen = FED. C FED. (USA) FD. FED. r o

Users' consent - simple as SAML David Simonsen = FED. C FED. (USA) FD. FED. r o Kalmar Kalmar FED. Kalmar s s f e d FED. g e e d g e w l d n o w l e K o e K n g e e d w l n o K r FED. a e n g

480 views • 21 slides

STUDY OF BALLOT QUESTION PROCESS League of Women Voters of Massachusetts Fall 2018 Winter

1 STUDY OF BALLOT QUESTION PROCESS League of Women Voters of Massachusetts Fall 2018 Winter 2019 2 Study Goals Educate LWV members about the ballot question process in MA At each League, consider, through consensus meetings,

581 views • 24 slides

Retail Recovery Playbook PAGE 1 Retail Council of Canada, in partnership with the Boston

Retail Recovery Playbook PAGE 1 Retail Council of Canada, in partnership with the Boston Consulting Group are leveraging global best practices for a diverse range of retail formats with the fundamental aim of ensuring a safe environment for

494 views • 3 slides

Getting Started with Azure SQL Database CodeStock April 12, 2019 Who is Chad Green Director

Chad Green Getting Started with Azure SQL Database CodeStock April 12, 2019 Who is Chad Green Director of Software Development at ScholarRx Community Involvement Code PaLOUsa Conference Chair Louisville .NET Meetup Organizer

680 views • 29 slides

Sc Scar ary Af y Afte ter All r All Boar ard Fu d Fundr drai aisi sing g Tr Trai

Fundraising Isnt So Sc Scar ary Af y Afte ter All r All Boar ard Fu d Fundr drai aisi sing g Tr Trai aining g Worksh shop @fundraiserchad AGENDA: 1. Introductions 2. Fundraising 101 / Board Role 3. Telling Our Story 4. The

1.09k views • 73 slides