From Research T o T ransfer: User-Controllable Cloud Storage - - PowerPoint PPT Presentation

Department of Computer Science | Institute of Systems Architecture | Chair of Computer Networks

From Research T

• T

ransfer: User-Controllable Cloud Storage

14th TERENA TF-Storage Meeting February 10-11, 2014, Zurich, Switzerland

Josef Spillner mailto:josef.spillner@tu-dresden.de

xmpp:josef.spillner@jabber.org

The T rouble with Cloud Storage

3 months limit; tariff choice utility billing backup/sync

• vs. storage

web access

• nly

proprietary client file size limit institutional pricing: progressive/ degressive trust? quality?

T aming the T rouble: RAICs, RAOCs

controller/ gateway

storage targets: services, devices service properties service bundles auto-selection auto-configuration auto-repair integration scheduling

• ptimisation

proxy cloud extension backup sync sharing ...

The Cloud Storage Lab @ TU Dresden

prototypical software development research experiments and pilot installations transfer widespread production deployments Short history:

• 2010 ideas and service modelling
• 2011 NubiSave storage controller started

UCC paper on Cloud storage controllers

• 2012 Usable software, Debian packaging
• 2013 NubiVis; π-Box personal Cloud VM

FGCS article on optimal storage SUCRE summer school tutorial at KIT

• 2014 NubiGate VM

http://lab.nubisave.org/

Research

Storage Service Ecosystem

Excursus: Service Science

• > Bundles -> RAOC

domain 1 domain 2 domain Cloud Storage

service interface functionality non-functional properties

nfp 1 nfp 2 nfp backup super-domain Infrastructure sub-domain File Storage

protocol description service instance service hosting client consumer provider service impl engineer SLAs messages

Storage Service Descriptions

webService GoogleStorage importsOntology { _"urn:ontology:conqo/CloudQoS.wsml#" } capability ServiceCapability postcondition definedBy ?serviceType memberOf cloud#CloudStorage . instance PricePerData memberOf { cloud#PricePerData, qos#ServiceSpec } qos#value hasValue 0.17 qos#unit hasValue qos#Euro concept MeasurementUnit conversionFactor impliesType _double concept SpaceUnit subConceptOf MeasurementUnit instance GB memberOf SpaceUnit conversionFactor hasValue 1024.0

Base Ontology (WSML) Instance Ontology

Storage Service Bundles

Service bundle $$ x a

(k+m) erasure coding secret sharing

s1 s2 s3 s4 s5 s6 s1 s2 s3 s4 s5 s6

replication

s1

Storage Service Markets

Storage Flows

Storage Targets Data Flow Integration

local directory USB stick NFS export WebDAV Commercial (Dropbox) local directory CIFS proxy Splitter/ Multiplexer File modifier (enc,dedup) block modifier (compress) block modifier (stego) Splitter/ Multiplexer Splitter/ Multiplexer Splitter/ Multiplexer web application sink source transport file modifier (enc)

Reconfjgurable Storage Flows

... alias Software-Defined Storage Modules (i.e. FUSE modules):

• Splitter (1:n)
• Modifiers (1:1)

* encryption * compression * ...

• Transports

* Directory * CIFS, NFS * WebDAV * SSHfs * S3fs * ... * CloudFusion

• SDS

propagation

Multi-User Storage Gateways

FlexDDPL: Flexible Data Distribution Policy Language scopes → applied in contexts (e.g. user database) ~user !negation T:22:00-06:00 @group fragment mime:message/rfc822 rules → applied to targets (e.g. storage provider) store control adapt

Feature: Algorithms Selection

Background

• secret sharing
• forward error correction / replication

Striping (→ RAID0)

• capacity 1.0, safety 1.0, performance 2.0

Mirroring (→ RAID1)

• capacity 0.5, safety 2.0, performance 1.0

XOR parity (→ RAID5, RAID6)

• 4 disks: capacity 0.75, safety 1.33, performance 1.0

Erasure coding (→ RAIDn)

• n storage targets, k significant, m redundant: capacity k/m, safety 1+m/k
• (Rotated) Reed-Solomon:

maximum-[Hamming-]distance separable (MDS) property

• Cauchy-Reed-Solomon, Vandermonde-Reed-Solomon:

practically secure; CRS faster due to XOR operations compared to GF(2w) AONT: information-theoretically secure AONT-RS: blend of AONT with (C)RS; alternative: encrypted fragments

Feature: Nested Contexts

context »normal« context »private«

Configuration Data

Feature: Weights

Assigning weights to the storage targets

• to fully utilise the capacity
• to exploit faster upload connections

Feature: Recursion

Saving the database which contains all file and fragment metadata...

• to avoid single point of failure
• to allow for selective sharing

Feature: Streaming (1:x)

Continuous operation during read or write of large file

• buffers on controller device can be kept small
• improved performance through parallel coding and transmission

Feature: Batching (x:1)

1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000

Encoding speed of different batch sizes

15 nodes

M=0 M=4 M=8 M=12

Filechunk size [KB] Encoding [MBs]

Experiments

peaCS: Performance and Efficiency Analsis for Cloud Storage

Nubisave Read Performance - 1MB File - UseAllInParallel Read in KB/s 1 2 3 4 5 6 7 8 9 10 Storages 0 10 20 30 40 50 60 70 80 90100 Redundancy 4000 6000 8000 10000 12000 14000 5000 6000 7000 8000 9000 10000 11000 12000 13000

Experiment Week Feb 7-14 2014

Autonomous Storage Clients

Scenario: A robot wants to store some data...

Labcast available: http://www.youtube.com/ watch?v=NDlN0fG9Okk

Autonomous Storage Clients

Scenario: Pervasive storage on all devices

Credentials may be auto generated

Software

Comparison of Storage Controllers

Prototype Origin Authors Approach Integration Public Availability Dependable Sky ('11)

T-Clouds,

• Univ. of Lisbon

Bessani, Corr- eia, Quaresma, André, Sousa data units, byzantine fail, Rabin erasure application library no

('11)

HPI, Potsdam Maxim Schnja- kin, Christoph Meinel NFPs, propr. platform, Libe- ration erasure web interface no

TrustedSafe ('10)

eGovCD, Berlin/ FhG FOKUS (no publication) n fragments:

• ne on USB,
• thers remote

Windows plugin no

Cloud Shredder ('11)

Chinese Academy of Sciences Nan Zang, Jiwu Jing, Peng Liu Two fragments:

• ne local, one

remote (not yet known) no

SecCSIE ('11)

T-Systems MMS, Dresden Ronny Seiger, Stephan Groß Cachy Reed- Solomon erasure CIFS proxy no

RACS ('10)

Cornell University Abu-Libdeh, Princehouse, Weatherspoon erasure coding: Zfec Reed- Solomon S3 proxy, can be distributed w/ Zookeeper yes (www.cs.cornell .edu)

plus numerous commercial implementations: NetApp StorageGRID, Cleversafe, Trusted Safe etc.

since 2012, community projects: Tahoe-LAFS externals, TCD, OpenStack/Intel - PyEC2Lib

OpenStack Intermezzo

NubiSave Cloud Storage Controller

Installation of NubiSave

• Git repository
• git://nubisave.org/git/nubisave
• Debian packages
• http://nubisave.org/packages/
• Integrated into π-Box or NubiGate VMs
• http://nubisave.org/downloads/

NubiSave Storage Flow Editor

NubiSave Confjguration

NubiSave Confjguration

NubiSave T

• ols

Controller operation $ n u b i s a v e [ < i n s t a n c e > ] Master script which starts both

• f the below combined.

$ n u b i s a v e h e a d l e s s Starts a new instance of the splitter/dispersion file system. $ n u b i s a v e g u i Starts the storage flow editor with storage integration configuration. $ n u b i s a v e s t

• p

Stops the splitter. Storages and database $ n u b i s a v e

• s

t a t u s Lists all splitter instances and storage targets attached to them

• r as part of any storage flow.

$ n u b i s a v e

• m
• u

n t e r [ < m

• d

u l e > ] Mounts all splitters and/or storage targets. $ n u b i s a v e

• u

n m

• u

n t e r [ < m

• d

u l e > ] Unmounts; inverse of the above. $ n u b i s a v e

• d

a t a b a s e [ . . . ] Inspection of storage metadata.

Splitter-NG Framework

Plugins: Jerasure, JSharing, RAID-1, more coming...

T ransport Module: CloudFusion

Additional Implementations

NubiVis: Distributed Data Visualisation

NubiVis Map View

NubiGate VM: Everything in one Box

NubiGate VM: OwnCloud Frontend

Our instance „RN-Dropbox“ 141.76.41.232/owncloud Your instance...? 141.76.41.206 141.76.41.207

Future Vision

Future Work

What's next in our lab

• Software improvements
• Configuration through the web interface (student thesis)
• Git-based service registry query
• Research: MapReduce-like processing over dispersed data
• Scaling large with more users and storage providers, HA setups etc.
• Federation between research institutions:

reciprocative/collaborative storage resource sharing <-- role of NRENs

Publications

[SS13] Josef Spillner, Alexander Schill: Orchestration of Distributed Storage Targets through Storage Flows. 5th IEEE International Conference on Cloud Computing Technology and Science (CloudCom), Bristol, UK, December 2013. [SQ+13] Josef Spillner, Maximilian Quellmalz, Martin Friedrich, Alexander Schill: peaCS - Performance and Effjciency Analysis for Cloud Storage. Workshop on Cloud Storage Optimisation (CLOUSO) @ 2nd ESOCC, Málaga, Spain, September 2013. [SMS13] Josef Spillner, Johannes Müller, Alexander Schill: Creating Optimal Cloud Storage Systems. Elsevier Future Generation Computer Systems (FGCS), Issue 29(4), p. 1062-1072, June 2013. DOI 10.1016/j.future.2012.06.004. [SS12] Josef Spillner, Alexander Schill: Flexible Data Distribution Policy Language and Gateway Architecture. 1st IEEE Latin American Conference on Cloud Computing and Communications (LatinCloud), Porto Alegre, Rio Grande do Sul, Brazil, November 2012. [SP+12] Josef Spillner, Christian Piechnick, Claas Wilke, Uwe Aßmann, Alexander Schill: Autonomous Participation in Cloud Services. 2nd International Workshop on Intelligent Techniques and Architectures for Autonomic Clouds (ITAAC) @ 5th IEEE/ACM UCC, Chicago, Illinois, USA, November 2012. [SG+11] Josef Spillner, Gerd Bombach, Stefgen Matthischke, Johannes Müller, Rico Tzschichholz, Alexander Schill: Information Dispersion over Redundant Arrays of Optimal Cloud Storage for Desktop Users. 4th IEEE/ACM International Conference on Utility and Cloud Computing (UCC), Melbourne, Australia, December 2011. [Best Paper Award]