Where to store all the IoT Data? Piotr Robert Konopelko Business - - PowerPoint PPT Presentation

Where to store all the IoT Data?

Piotr Robert Konopelko Business & Technical Support Manager, MooseFS Pro piotr.konopelko@moosefs.pro

Linux Servers + MooseFS = Storage

High Availability High Performance Petabytes Longterm

What is MooseFS Pro?

MooseFS Pro is a software product: fault tolerant, network distributed file system It distributes your data over many servers acting as one large disk.

Architecture

Advantages

Mission Critical

The MooseFS Software-Defined Storage is designed for mission critical applications with high availability and high performance requirements.

High Availability

No Single Point of Failure (SPOF-less configuration). Metadata of the file system is kept in two or more copies on physical redundant servers. User data is redundantly spread across the storage servers in the system.

Advantages

Scalability

• Adding new and removing old nodes on the fly in minutes
• New space available immediately
• Storage can be extended up to 16 Exabytes

with more than 2 billion of files.

Advantages

High Performance

Designed to support high performance I/O operations. User data is read/written in parallel, directly on many storage nodes at once avoiding single central server or single network connection bottlenecks.

Big Data

The MooseFS storage software is designed for Big Data support. MooseFS enables virtually limitless pool of storage to support the most demanding distributed workloads.

Advantages

Hardware / Platform Independence

All-Flash and Hybrid storage setup supported. Different manufacturers' disks and servers may be used in the one storage system. Older and newer technology may be mixed if required. MooseFS can be ran on wide range of clusters from Raspberry Pi to Enterprise Servers on many architectures

Advantages

Commodity Hardware Components

The MooseFS based storage is built with commodity components of virtually any manufacturer. It supports all major disks and disk interfaces types: SATA / SAS, SSD / HDD.

Manufacturer Support

Support you get comes directly from the software manufacturer. We know each line of the source code, we can solve each issue.

Advantages

Lifetime Usability

Open source software with commodity components and operating systems make MooseFS safe and usable for entire lifetime.

Longer Hardware Life

Tiering and rolling upgrade features make hardware life longer as older, smaller and slower servers and/or disks may be used in less intensive tiers.

Use Cases

• IoT Data

– Sensors – Control Systems – Data Acquisition Equipment

• Surveillance
• Medical Data
• Scientific Data
• Clustering
• Hosting, Virtualization, HCI
• Big Data
• Supercomputing
• Backup and Archiving

Success Stories

Success Story 1

Country: Warsaw, Poland Market: Media & Entertainment, Market analysis Company: Medium company with 20+ European offices

Purpose of use: MooseFS is used as a primary storage for Internet traffic data measurement, which is the core business data for this company. MooseFS is installed on a few separate storage clusters, where heavy calculations are performed along with storing data. Concurrently used by 300+

• nline users and many background data processing applications. Clusters receive hundreds of

thousands of new records per second. Clusters have been online since 2005 and are built with over 150 servers storing a few petabytes in total.

Success Stories

Success Story 2

Country: Boston, MA, USA Market: Healthcare, Education, Research Company: One of the prominent Ivy League Universities

Purpose of use: MooseFS is used on two clusters in the university labs: the first is designed for medical data storage and processing, while the second is used to store VMs disks' images. A few dedicated custom features were added to the system for this customer. Clusters have been online since 2013 storing half a petabyte of crucial data.

Success Stories

Success Story 3

Country: New York, USA Market: Research, Education Company: One of the prominent Ivy League Universities

Purpose of use: MooseFS is used as a storage solution for human genome research activities. Interesting pattern of MooseFS’s atomic snapshot feature use case: creating and discarding genome data for different proposals of research algorithms. More than 10 Petabytes of data is stored on a single cluster.

Success Stories

Success Story 4

Country: Sweden Market: Cloud & Hosting Company: Cloud Provider

Purpose of use: MooseFS is used as storage for on-premise cloud solutions for their customers. Around 30 servers and 50 end-application connections creating several Petabytes storage cluster. The solution includes storage for surveillance systems.

Success Stories

Success Story 5

Country: France Market: Research Company: European oceanographic research institute

Purpose of use:

MooseFS is used as a primary storage for research and data analysis for the exploration

• f the sea. Computations on the data are moved close to the data and run on the same

machines where data is stored. Nearly half a billion files, mostly satellite images consuming around 4 petabytes of data, stored on over a hundred servers and serving almost a few hundred computing applications.

Success Stories

Success Story 6

Country: Poland Market: Cloud & Hosting Company: Data Center Storage, CDN

Purpose of use:

MooseFS is used as a primary distributed storage in one of their Data Centers in

• Poland. Two clusters: one used as a backend storage for internal projects, on-premise

hosting and Content Delivery Network (images and videos mainly) for company's customers, second one is based on MooseFS + Proxmox integration and provides VPS

• hosting. Storing a few petabytes of data in total and counting.

Success Stories

Success Story 7

Country: Middle East Market: Cloud & Hosting Company: CDN, Cloud & Hosting

Purpose of use: MooseFS is used as a core backend storage solution for CDN clusters (images, ringtones, and videos). A few hundreds of millions of files store a couple

• f petabytes of data in total. The company is a leading Content Delivery

Network Provider.

Success Stories

Success Story 8

Country: Warsaw, Poland Market: Market: Media & Entertainment, Internet television Company: A division of the biggest Polish media group

Purpose of use: MooseFS is used as a backend of the Content Delivery Network (CDN) service for internet TV serving content in Video on Demand manner. Interesting case

• f transcoding video on the same servers where data is stored. Over 30 servers

are used to store a few Petabytes of video and image content.

Technical Features

Redundancy

All system components are redundant and in case of failure there's an automatic failover mechanism that is transparent to the users.

Compute on Nodes

Support for scheduling computation on data nodes for better overall system TCO by utilizing idle CPU and memory resources.

Technical Features

Rolling Upgrades

Ability to perform one node at a time system upgrades without service disruption including hardware replacement and additions. This feature allows to maintain hardware platform up-to-date with no downtime.

Tiered Storage

All system components are redundant and in case of failure there's an automatic failover mechanism that is transparent to the users.

Technical Features

Fast Disk Recovery

Ability to perform one node at a time system upgrades without service disruption including hardware replacement and additions. This feature allows to maintain hardware platform up-to-date with no downtime.

Atomic Snapshots

Instant, uninterrupted provisioning of the state

• f the filesystem at a particular point in time.

This feature is ideal for on-line backup solutions.

Technical Features

POSIX Compliance

Support for family of standards, specified by the IEEE, to clarify and make uniform the application programming interfaces (and ancillary issues, such as command line shell utilities) provided by Unix like operating systems.

Erasure Coding (”Distributed RAID”)

Ensuring data redundancy using error correction code algorithms with up to 9 parities. It saves raw data space compared to an ordinary data duplication approach.

Technical Features

Global Trash

A virtual, global space for deleted objects configurable for each individual file and directory. A very useful feature for recovering accidentally deleted data caused by human error.

Quota Limits

Limits set by a system administrator to restrict certain aspects of file system usage and to allocate limited disk space in a reasonable way, i.e. number of i-nodes

• r capacity on directory level.

Technical Features

Access Control

Access to files and directories is based on a standard Unix access control model enhanced with standard Access Control Lists.

Native Clients

For performance reasons there is a dedicated client component for Linux, FreeBSD and Mac OS X systems.

Technical Features

Parallelism

Performs all I/O operations in parallel threads of executions to deliver good read/write operations performance.

Ethernet

1-200 Gbps standard Ethernet based network used for all the communication with support for LACP configurations.

Technical Features

Hardware Independence

Works out-of-the-box on almost any hardware platform that can run POSIX compliant Operating System like Linux, macOS or FreeBSD.

Thank you

peter@moosefs.pro moosefs.pro