PDSW 2020:5TH INTERNATIONAL PARALLEL DATA SYSTEMS WORKSHOP Fi Fingerprinting t g the C Check cker Po Policies of Pa Parallel File Systems Runzhou Han , Duo Zhang, Mai Zheng
Parallel File Systems (PFSes) • PFS is the cornerstone of high performance computing • Optimized for highly concurrent access 1
PFS Failures: Real-World Cases Case1: HPCC Power Outage 2
PFS Failures: Real-World Cases Case1: HPCC Power Outage Case2: ACCRE Storage Outage* 3 * Hyperion Research survey of HPC organizations done for Panasas
PFS Failures : More Frequent/Expensive Than You Thought Some statistics *: The average HPC storage system failure frequency is 9.8 failures /year ≈ half of HPC sites experience storage system failures 1/month or more frequently 4 * Hyperion Research survey of HPC organizations done for Panasas
PFS Failures : More Frequent/Expensive Than You Thought Some statistics *: The average HPC storage system failure Downtime ranges from frequency is 1 day ↓ to 1 week ↑ 9.8 failures /year ≈ half 40 % of HPC sites experience storage of HPC sites typically took system failures more than 2 weeks to restore 1/month or more their storage systems frequently 5 * Hyperion Research survey of HPC organizations done for Panasas
PFS Failures : More Frequent/Expensive Than You Thought Some statistics *: The average HPC A single day of downtime storage system failure Downtime ranges from costs from frequency is 1 day ↓ to 1 week ↑ $100K ↓ to $1M ↑ 9.8 failures /year ≈ half 40 % of HPC sites Average downtime cost is experience storage of HPC sites typically took $127K /day system failures more than 2 weeks to restore 1/month or more their storage systems frequently 6 * Hyperion Research survey of HPC organizations done for Panasas
PFS & PFS Checkers (FSCKs) • Typical PFS architecture Network Management Metadata Object Storage Server (MGS) Server (MDS) Servers (OSSes) Object Storage Management Metadata Targets (OSTs) Target (MGT) Target (MDT) 7
PFS & PFS Checkers (FSCKs) • Typical PFS architecture Network Management Metadata Object Storage Server (MGS) Server (MDS) Servers (OSSes) Object Storage Management Metadata PFS checker Targets (OSTs) Target (MGT) Target (MDT) (FSCK) I/Oes I/Oes I/Oes • Many PFSes are designed with a checker component • e.g., LFSCK for Lustre, BeeGFS-FSCK for BeeGFS, PV2FS-FSCK for OrangeFS 8
PFS & PFS Checkers (FSCKs) • Typical PFS architecture Network Management Metadata Object Storage Server (MGS) Server (MDS) Servers (OSSes) Object Storage Management Metadata PFS checker Targets (OSTs) Target (MGT) Target (MDT) (FSCK) I/Oes I/Oes I/Oes • Many PFSes are designed with a checker component • e.g., LFSCK for Lustre, BeeGFS-FSCK for BeeGFS, PV2FS-FSCK for OrangeFS • Detect and repair inconsistencies 9
PFS & PFS Checkers (FSCKs) • Typical PFS architecture Network Management Metadata Object Storage Server (MGS) Server (MDS) Servers (OSSes) Object Storage Management Metadata PFS checker Targets (OSTs) Target (MGT) Target (MDT) (FSCK) I/Oes I/Oes I/Oes • Many PFSes are designed with a checker component • e.g., LFSCK for Lustre, BeeGFS-FSCK for BeeGFS, PV2FS-FSCK for OrangeFS • Detect and repair inconsistencies • FSCKs have predefined checker policies 10
Examples of PFS Checker Policies • Lustre’s LFSCK Policy: mapping between MDT-object and OST-object • MDT-object’s LOV EA matches to OST-object’s FID • OST-object’s Parent FID matches to MDT-object’s FID MDT OST Structures Meaning xattr xattr xattr inode extended attribute MDT- OST- FID FID objects objects data FID a global ID of an Lustre object LOV EA Parent LOV EA stores child object’s FID FID … MDT-object A Parent FID stores parent object’s FID OST-object a 11
Examples of PFS Checker Policies • Lustre’s LFSCK Policy: mapping between MDT-object and OST-object • MDT-object’s LOV EA matches to OST-object’s FID • OST-object’s Parent FID matches to MDT-object’s FID MDT OST Structures Meaning xattr xattr xattr inode extended attribute MDT- OST- FID FID objects objects data FID a global ID of an Lustre object LOV EA Parent LOV EA stores child object’s FID FID … MDT-object A Parent FID stores parent object’s FID OST-object a 12
Examples of PFS Checker Policies • Lustre’s LFSCK Policy: mapping between MDT-object and OST-object • MDT-object’s LOV EA matches to OST-object’s FID • OST-object’s Parent FID matches to MDT-object’s FID MDT OST Structures Meaning xattr xattr xattr inode extended attribute MDT- OST- FID FID objects objects data FID a global ID of an Lustre object LOV EA Parent LOV EA stores child object’s FID FID … MDT-object A Parent FID stores parent object’s FID OST-object a 13
Examples of PFS Checker Policies • Lustre’s LFSCK Policy: mapping between MDT-object and OST-object • MDT-object’s LOV EA matches to OST-object’s FID • OST-object’s Parent FID matches to MDT-object’s FID MDT OST Structures Meaning xattr xattr xattr inode extended attribute MDT- OST- FID FID objects objects data FID a global ID of an Lustre object LOV EA Parent LOV EA stores child object’s FID FID … MDT-object A Parent FID stores parent object’s FID OST-object a 14
Examples of PFS Checker Policies • Lustre’s LFSCK Policy: mapping between MDT-object and OST-object • MDT-object’s LOV EA matches to OST-object’s FID • OST-object’s Parent FID matches to MDT-object’s FID MDT OST Structures Meaning xattr xattr xattr inode extended attribute MDT- OST- FID FID objects objects data FID a global ID of an Lustre object LOV EA Parent LOV EA stores child object’s FID FID … MDT-object A Parent FID stores parent object’s FID OST-object a 15
Examples of PFS Checker Policies • Lustre’s LFSCK Policy: mapping between MDT-object and OST-object • MDT-object’s LOV EA matches to OST-object’s FID • OST-object’s Parent FID matches to MDT-object’s FID MDT OST Structures Meaning xattr xattr xattr inode extended attribute MDT- OST- FID FID objects objects data FID a global ID of an Lustre object LOV EA Parent LOV EA stores child object’s FID FID … MDT-object A Parent FID stores parent object’s FID OST-object a 16
Examples of PFS Checker Policies • Lustre’s LFSCK Policy: mapping between MDT-object and OST-object • MDT-object’s LOV EA matches to OST-object’s FID Corruption 1 • OST-object’s Parent FID matches to MDT-object’s FID MDT OST Structures Meaning xattr xattr xattr inode extended attribute MDT- OST- FID FID objects objects data FID a global ID of an Lustre object LOV EA corruption Parent LOV EA stores child object’s FID FID … MDT-object A Parent FID stores parent object’s FID OST-object a 17
Examples of PFS Checker Policies • Lustre’s LFSCK Policy: mapping between MDT-object and OST-object • MDT-object’s LOV EA matches to OST-object’s FID Fixed! Corruption 1 • OST-object’s Parent FID matches to MDT-object’s FID MDT OST Structures Meaning xattr xattr xattr inode extended attribute MDT- OST- FID FID objects objects data FID a global ID of an Lustre object ✔ LOV EA LFSCK Parent LOV EA stores child object’s FID FID … MDT-object A Parent FID stores parent object’s FID OST-object a 18
Examples of PFS Checker Policies • Lustre’s LFSCK Policy: mapping between MDT-object and OST-object • MDT-object’s LOV EA matches to OST-object’s FID • OST-object’s Parent FID matches to MDT-object’s FID Corruption 2 MDT OST Structures Meaning xattr xattr xattr inode extended attribute MDT- OST- corruption FID FID objects objects data FID a global ID of an Lustre object LOV EA Parent LOV EA stores child object’s FID FID … MDT-object A Parent FID stores parent object’s FID OST-object a 19
Examples of PFS Checker Policies • Lustre’s LFSCK Policy: mapping between MDT-object and OST-object • MDT-object’s LOV EA matches to OST-object’s FID • OST-object’s Parent FID matches to MDT-object’s FID Cannot be fixed! Corruption 2 MDT OST Structures Meaning xattr xattr xattr inode extended attribute MDT- OST- ✘ FID LFSCK FID objects objects data FID a global ID of an Lustre object LOV EA Parent LOV EA stores child object’s FID FID … MDT-object A Parent FID stores parent object’s FID OST-object a 20
Examples of PFS Checker Policies • Lustre’s LFSCK Policy: mapping between MDT-object and OST-object • MDT-object’s LOV EA matches to OST-object’s FID • OST-object’s Parent FID matches to MDT-object’s FID Cannot be fixed! Corruption 2 LFSCK’s policy is incomplete! MDT OST Structures Meaning xattr xattr xattr inode extended attribute MDT- OST- ✘ FID LFSCK FID objects objects data FID a global ID of an Lustre object LOV EA Parent LOV EA stores child object’s FID FID … MDT-object A Parent FID stores parent object’s FID OST-object a 21
Our Contributions • A systematic approach to analyze PFS checker policies • PFS type-aware fault injection • PFS consistency model & taxonomy 22
Recommend
More recommend
