AXF – Archive eXchange Format: Interchange & Interoperability for Operational Storage and Long-Term Preservation Report to SMPTE Washington DC Section Bits By the Bay Conference 21 May, 2014 S. Merrill Weiss / Merrill Weiss Group LLC Chairman, TC-31FS30 AXF Working Group
Background • History – Proprietary Systems • Formats on Media • Interface Protocols • Same System Type Required for Recovery • Media Migration Not Easy – Danger of Orphaned Archives — If System Support Ended – High Costs of Implementation & Operation • Individualized System Integration Requirements • Transfer Costs Resulting from Inability to Interchange Media
Objectives • User Requirements – Movement of Material Between Different Archive Systems • Between Different Operations of Same Company • Between Companies • Retrieving Files & Metadata from Media Into Different Systems – Flexibility in Changing Archive Management Vendors • No Loss of Data or Metadata When Changing Vendors • No Requirement for Native Use of Format
Objectives • User Requirements – cont’d. – Archive Investment Protection • Ability to Retrieve Files & Metadata In Absence of Creating System • Ability to Read Media Into Other Archive Systems • Ability to Retrieve Media Contents w/Simple Utility on Many OSs – Automation Metadata Support • Inclusion of Metadata for Systems Interacting w/Archive Systems – E.g., Traffic, Automation, Color Correction, Editing Systems • Allow Importing “Discovered” Items into Databases
Objectives • Additional Requirements – Providing Unlimited Storage Capability • File Size, Number of Files, Number of Media, etc. – Providing an Implementation Strategy – Providing Support for All Types of Media – Current & Future – Providing for Storage on More Than One Medium – “Spanning” – Providing for Updates/Changes to Archive Objects over Time – Providing for Information Recovery from Damaged Media – Providing for File Exchange via Communications Channels • Enables Use of Cloud Storage
Objectives • Underlying Assumption – Same Type of Media Supported on Both Systems • Source of Medium & Recipient of Medium • Drives, Drivers, & Control Software
Implementation Strategy • Large Existing Installed Base & Large Archive Inventory • Initially, Explicit Export & Import – Export of Archive Objects to Media Specifically for Interchange – Importing of Archive Objects into Receiving Systems • By Translating to Native Formats of Receiving Systems • By Inclusion of Interchanged Objects in Databases • Later, Adoption of Format as Native in Archive Systems – Eliminating Need for Separate Export/Import Steps – Permitting Direct Transfer of Media Between Systems
Implementation Strategy • Data Recovery Utilities Available Through SMPTE – To Be Contributed by WG Participants – For Wide Variety of Operating Systems – For All Media Types – Permitting Data Recovery without an Archive System – To Help Ensure Access to Archived Files & Associated Metadata
SMPTE Standard • SMPTE ST 2034 Part 1 Written & In Ballot – Result of Years of Work & Refinement – Uses XML Schema to Define Most Structures – Expected to Be Completed This Year • Part 2 Will Be Standard on Use of AXF Schema Upstream – Providing Mechanism for Communicating Metadata to Archive Systems – Most Major Functions Defined – Writing of Document Begun
How Are Digital Assets Stored? • Steps Required to Ensure Long-Term Accessibility – Valuable File-Based Assets Stored in Digital Archives in All Industries • Key Goals of the “Ideal Storage Format” – Ensuring Long Term Accessibility – Self-Describing Assets & Self-Describing Storage Media – Encapsulation to Maintain Important Metadata & File Relationships – Scalability for Any Number of Elements of Any Size & Type – Standardized Regardless of Storage Media Technology – Transportability & Compatibility between Systems – Preservation (OAIS) Features such as Fixity, Provenance, etc.
Existing Storage Choices • TAR – Tape ARchive Format (Originally from UNIX) • Has Been Around for Many Decades – No Truly Universal TAR “Standard” • Rather, Many “Customized” Implementations – TAR Is a Legacy Format • Cannot Support Many Core Functionalities Required in M&E Space – TAR Misses Many of Storage Format Goals Outlined
Existing Storage Format Choices • LTFS – Linear Tape File System is Simple File System for Linear Data Tape – Makes Data Tapes Appear as “Removable Storage” – Currently No Standards Bodies Have Documented LTFS • Often Mistakenly Referred to as a “Standard” • Being Documented by Storage Networking Industries Association – Considered By Itself, Has Some Significant Limitations • With Respect to Storage Format Goals Outlined – By Itself, Is Very Useful for Physical “Transport” of Content • But Not for Long Term Storage or Preservation – Why Not?
Existing Storage Format Choices • LTFS cont’d. – No Media Encapsulation • Relies on Simple Folder Hierarchies to Form Important Asset Relationships • Lacks Context – Does Not Scale Well • Due to Lack of Support for Spanning Across Storage Media – A Significant Problem in M&E – Only Supports Modern Data Tape Technologies • Is Not Applicable to Any Other Storage Technologies
Existing Storage Format Choices • TAR & LTFS – Neither Achieves 100% of Long-Term Storage & Preservation Goals • What other choices are there? • Caution: Following Detailed Descriptions Are Not Adopted Yet – Some Details Still May Change in Response to Ballot Inputs
AXF – Archive eXchange Format • AXF – A File Collection “Wrapper” – Essentially an Advanced ZIP – Can Encapsulate Any Number of Files of Any Type & Size • Brings Universal Transport & Interoperability to Storage – At Same Level as MXF Brings to Content • Designed to Support All Storage Technologies – Now and into the Long-Term Future • IT‐Centric and Not Tied to Media Applications Alone • Being Standardized by SMPTE
AXF Layered Context • Considered within an IT System Stack Server/Storage Stack with AXF support AXF-Aware Application Archive eXchange Format (AXF), Including Internal File System Block Level Addressing File System Operating System – Hardware Abstraction Layer Driver Physical Drive Medium (with File System) Medium (without File System)
AXF Features • Unlimited Storage Support – Any Number of Files, Any Size of Files, Plus Media Spanning • Resilience to Media Damage and Corruption – Redundancy in All Structures – Payload Independently Recoverable • Support for Media With & Without a File System – Raw Data Tape, LTFS Data Tape, Spinning Disk, Flash Media, Optical Storage, Holographic Storage, & Anything in Future • Support for Any Operating System and/or File System – Embedded File System Abstracts Underlying Systems
AXF Features • Self Describing Objects & Self Describing Media – Enable Simple Transport of Objects & Media between Systems • Object Versioning & Collection Support – Support Complex Relationships between Objects – Support Additions, Updates, etc. • Support for All File Types – Not Just Media Files – IT‐Centric Implementation • Based on Experience Handling Big Data in M&E
AXF Features • Streaming & File-Based Asset Transport & Delivery – Support for Streaming De‐Encapsulation – Support for In‐Path Checksums for Structures & Files – Support for Cloud Storage & Delivery Applications – Streaming Design Maximizes Transfer Speeds • To/from All Media Types
What Is AXF? File Ancillary Metadata Payload Files Structured A V Unstructured Asset Components Proprietary A V Open Preservation Access Provenance Fixity Context Reference Elements Control File Universal Storage-Agnostic File System System AXF Object
Storage Technology & File Systems Future Storage Technologies Spinning Disk Holographic (NTFS, FAT , etc.) (UDF , etc.) AXF Object Solid State Disk Blu-Ray (NTFS, FAT , etc.) (BDFS, UDF , etc.) Flash Media Data Tape DVD (NTFS, FAT , etc.) (Block-Based, LTFS, etc.) (UDF , etc.)
AXF Medium AXF Object Identifier Header Metadata Metadata Container Payload AXF Object … 1 Metadata AXF on Storage Medium Container File 1 AXF Object AXF File Footer 2 File 2 Payload File AXF File Footer … … File N AXF Object N AXF File Footer AXF Object Footer
Inside an AXF Object • Binary Structure Containers Define Elements
Inside an AXF Object (2) • File Tree Acts as Light-Weight File System – Contained in Object Header & Footer – Identifies Payload Files, Locations, Sizes, & Other Characteristics
Recommend
More recommend
