[PPT] - Amalgamating Manila and Swift for Unified Data Sharing Across PowerPoint Presentation

SLIDE 1

Amalgamating Manila and Swift for Unified Data Sharing Across Instances

Bill Owen

Sr Software Engineer @wwowen

Sandeep Patil

STSM @Sandeep_r_patil

Brian Nelson

Software Engineer

Simon Lorenz

Software Engineer @SimLorenz

Gaurang Tapase

Software Engineer @gaurangtapase

SLIDE 2

Problem

SLIDE 3

Media Production House or Professional Sports Team

1. Employees collects videos and upload to a central object storage system 2. Employees edit videos using sophisticated image editing software. This software requires file access to these images 3. Each employee has access to data in dedicated container from object interface. 4. Same access credentials are used to access data from file interface 5. Requirement for multi-tenant support to segregate clients 6. Must be able to easily scale up compute based on priority and schedule of projects

Problem Scenario

Ingest data from Object interface, analyze data from file interface

SLIDE 4

Solution

SLIDE 5

1. Solution architecture to address these and similar use cases
2. Demo of sample deployment
3. Challenges to overcome to realize this architecture

Agenda

SLIDE 6

OpenStack Swift is the highly available, distributed object

storage system used for uploading unstructured data

SwiftonFile diskfile backend implementation for Swift enables
bject data to be accessed as files
OpenStack Manila is the file share service used to share file

data across VM instances. Multi-tenancy in Manila can help share data securely across tenants

Shared Filesystem backend for both Swift and Manila enables

common data access

OpenStack Keystone, an authentication service coupled with

AD/LDAP as common authentication backend enforce authentication and data access authorization

OpenStack Nova, Cinder, Glance etc. for creation of virtual

machines used for creating VM farm which will be used to analyze data

OpenStack Cloud

VM Instances Shared Filesystem Swift (with SwiftonFile) Manila (File Share) Shared Data AD/LDAP Object Data Manila shares over

bject data

Nova Cinder Glance

Components of the solution

SLIDE 7

Highly scalable, enterprise class OpenStack Object storage
Rich and growing feature set
Extensible:
custom middleware
pluggable backend: diskfile
Swift and Amazon S3 protocol support

OpenStack Swift

SLIDE 8

But how do file applications access object data?

SLIDE 9

Origin: Red Hat contribution in April 2014
Allows you to:
Write an object, read as a file
Write a file, read as an object
Swift Diskfile Module: different modules for

different types of storage

Swift Storage polices: coexists nicely with

traditional storage policies

Clients can access data directly from file

interface

Swift On File

Swiftonfile diskfile Swiftonfile diskfile Spectrum Scale GlusterFS XFS

Openstack Swift

default diskfile

SLIDE 10

File and Object Access Policy Standard Policy

Swift on File Storage Policy

SLIDE 11

Swift on File Storage Object Layout

This object:

http://swift.example.com/v1/acct/cont/obj

Is located here with traditional Swift:

/mnt/swift/z1device7/objects/63773/ba2/f91d1e7550cd32822a17b00fa86d9ba2/ 1414045361.93852.data

But is located here with Swift On File:

/mnt/scaleout_fs/acct/cont/obj

SLIDE 12

What about easy remote access to this data? And access from VMs?

SLIDE 13

Shared file system service for OpenStack
Provides uniform interface for

management of file system shares from different storage backends

Many different vendor drivers today
Share Operations include:
Create, delete, extend, shrink shares
Create snapshot, create share from snapshot
Manage share access rules
Manage / unmanage existing data sets
Consistency groups, replication, …
Multi-tenancy support

Manila Overview

SLIDE 14

This is sounding better… But how do I share data between Swift on File and Manila?

SLIDE 15

Common data plane for OpenStack storage
Provides enterprise features like:
Snapshots & backups
Automatic tiering/migration of data across

storage pools

Local & multi-site Replication
Enables Unique Features for OpenStack

services

Nova live migration
Efficient data sharing between Glance, Cinder,

Nova using COW

Features vary between vendors, but the

idea is the same

Shared File System

Shared File Systems and OpenStack Storage

SLIDE 16

That seems simple enough…

SLIDE 17

Demo

SLIDE 18

Unified Authentication: Use same credentials for object interface and file interface Today:

Middleware to enable creation of objects with ownership (UID/GID) of requesting user

Future Extensions:

Mapping of Object ACLs to File ACLs
Mapping of Object quotas to File quotas
Define on either interface, enforce on both

Learn more at: “Write a File, Read as an Object” Thursday @ 1:30pm

But…

What are the challenges to make this work in a production environment?

SLIDE 19

More… Challenges to make this work in a production environment

Semantic Mismatch Object uses eventual consistency, file uses strong consistency Today:

Access managed by user & application workflow
Ingest through file, read through object
Ingest through object, analyze and publish through file, read results through object
Single writer to container or subdirectory at any point in time
Can be enforced by ACLs or export permissions
New files are automatically “objectized” by background daemon

Future Extensions:

Enforced coordination between object and file interfaces
Eventual goal: full simultaneous access from either interface

SLIDE 20

OpenStack provides the building blocks

for sharing data between wide variety

f clients and interfaces
An enterprise shared file system can be

the glue that brings the pieces together in the most efficient manner

Simple, hypothetical use cases

presented today

The infrastructure can be used to solve

the complex real life use cases that customers are facing today

OpenStack Cloud

VM Instances Shared Filesystem Swift (with SwiftonFile) Manila (File Share) Shared Data AD/LDAP Object Data Manila shares over

bject data

Nova Cinder Glance

Conclusions

SLIDE 21

Announcements

SLIDE 22

OpenStack & Spectrum Scale Red Paper Released this week:

SLIDE 23

SLIDE 24