Cloud File Services for Academics Harry Mangalam, RCS <harry.mangalam@uci.edu> x40084 Full Report: http://goo.gl/fCNSIa
BigData is here ● Most of what we see is unstructured data ● Or semi-structured (large ASCII files) ● 100TB datasets now on HPC ● More coming..
Bandwidth & Latency ● BW to many commercial services tops out at 1-5MB/s. ● Latency increases over distance and hops. ● Academic/Data networks can be 10-100X faster. ● LANs are fastest, often 100X faster than WANs. ● WANs block some protocols, ports, etc. ● Small file operations make everything worse.
Raw Storage Pricing ● Raw storage halves in price every ~14mo ● NAS-quality 4TB disk is $170 ($4.25/100GB) ● 250TB Chassis is $35K ($14/100GB) ● Lifetime of Chassis is 4yrs ($3.5/100GB)
Commercial Pricing ● Cheapest is ~$24/100GB/yr (Google, iCloud) ● Most expensive is ~$100/100GB/yr (some business services: Box, Druva) ● Pricing varies by business model (Backblaze is unlimited) ● Long Term Backup (Glacier @ $1/100GB/yr is cheapest but horrible interface. ● Since raw storage halves in price every 14mo, pricing changes very rapidly.
5 overlapping uses ● Interactive file storage ● File Sharing ● File Syncing (change to a file is propagated to all copies of that file) ● Short Term Backup (incr'l backup, disaster recovery, client-initiated recovery) ● Long Term Backup (dataset archives w. infrequent recovery)
Venn Diagram of previous slide
Important Features ● Ability to scale greatly. ● Support for very large files. ● Ease of user interface, informative ● Watch edits / co-editing. ● Easy to set/show permissions who can read, write, edit & delete
And More.. ● Pay only for what you use. ● Sync/autosync across lots of devices. ● Direct web-availability. ● File versioning like Apple’s TimeMachine or ZFS snapshots. ● High bandwidth. ● Highly reliable.
And Even More.. ● Multiple protocols (NFS, CIFS, HTTP/HTPPS, Globus, FTP/SFTP, etc). ● Mac/iOS/Win/Linux/Android clients ● Logging to show who has accessed files and how. ● Security (file sharing vs backups) ● Integrated Apps ● Stateless/Connectionless storage
Local vs Commercial ● Local resources tends to offer more, more flexible, faster services at a cheaper long-term cost with less lock-in. ● Commercial services tend to offer less HR, DC req, more stability, decent options, at a higher aggregate cost. ● Legal advantages to both. ● Admin and users often have different and often opposing requirements; Admin tends to make the final decisions. ● Best solution is probably “Both”.
Local Sharing Services ● OwnCloud – 'Open Source DropBox' Mac/iOS/Win/Linux/Android clients ● UCLA/CASS – Very large campus storage system, supports Globus/Grid, SMB/CIFS, NFS, CrashPlan.
Commercial Services & more.. ● See: <http://goo.gl/fCNSIa> ● Harry Mangalam <harry.mangalam@uci.edu> x40084
Recommend
More recommend
