Distributed File Storage in Multi-Tenant Clouds using CephFS - - PowerPoint PPT Presentation

Distributed File Storage in Multi-Tenant Clouds using CephFS

Openstack Vancouver 2018 May 23

Patrick Donnelly CephFS Engineer Red Hat, Inc. Tom Barron Manila Engineer Red Hat, Inc. Ramana Raja CephFS Engineer Red Hat, Inc.

How do we solve this?

Openstack Sydney: 2017 November 06

Ceph Higher Level Components

RGW

S3 and Swift compatible object storage with object versioning, multi-site federation, and replication

LIBRADOS

A library allowing apps to direct access RADOS (C, C++, Java, Python, Ruby, PHP)

RADOS

A software-based, reliable, autonomic, distributed object store comprised of self-healing, self-managing, intelligent storage nodes (OSDs) and lightweight monitors (Mons)

RBD

A virtual block device with snapshots, copy-on-write clones, and multi-site replication

CEPHFS

A distributed POSIX file system with coherent caches and snapshots on any directory

OBJECT BLOCK FILE

3

Openstack Sydney: 2017 November 06

• CephFS is a POSIX-compatible distributed

file system!

• 3 moving parts: the MDS(s), the clients, and

RADOS.

• Files, directories, and other metadata are

stored in RADOS. The MDS has no local state.

• Mount using:

○

FUSE: ceph-fuse ...

○

Kernel: mount -t ceph ...

• Coherent caching across clients. MDS issues

inode capabilities which enforce synchronous or buffered writes.

• Clients access data directly via RADOS.

What is CephFS?

4

Client Client ceph-mds

RADOS

Metadata RPC File I/O Journal Metadata

CephFS native driver deployment

Public OpenStack Service API (External) network Storage (Ceph public) network External Provider Network Storage Provider Network Router Router Tenant VMs with 2 nics Manila Share service Ceph MON Ceph MGR Ceph OSD Ceph OSD Ceph OSD Controller Nodes Storage nodes Tenant A Tenant B Compute Nodes Manila API service Ceph MDS placement: With MONs/python services/ dedicated? Ceph MDS

https://docs.openstack.org/manila/ocata/devref/cephfs_native_driver.html

CephFS NFS driver (in data plane)

Metadata updates Data updates

• Clients connected to NFS-Ganesha
• gateway. Better security.
• No single point of failure (SPOF) in Ceph

storage cluster (HA of MON, MDS, OSD)

• NFS-Ganesha needs to be HA for no SPOF

in data plane.

• NFS-Ganesha active/passive HA WIP

(Pacemaker/Corosync)

OpenStack client/Nova VM

Monitor Metadata Server OSD Daemon

server daemons NFS gateway

Native Ceph NFS

CephFS NFS driver deployment

Public OpenStack Service API (External) network Storage (Ceph public) network External Provider Network Storage NFS Network Router Router Tenant VMs with 2 nics Manila Share service Ceph MON Ceph MDS Ceph OSD Ceph OSD Ceph OSD Controller Nodes Storage nodes Tenant A Tenant B Compute Nodes Manila API service Ceph MDS placement: With MONs/python services/ dedicated?

https://docs.openstack.org/manila/queens/admin/cephfs_driver.html

Future: Ganesha per Tenant

Public OpenStack Service API (External) network Ceph public network External Provider Network Router Router Tenant VMs Manila Share service Ceph MON Ceph MDS Ceph OSD Ceph OSD Ceph OSD Controller Nodes Tenant A Tenant B Compute Nodes Manila API service Ceph MGR kubernetes

9

Kubernetes Pod (HA Managed by Kubernetes) MDS OSD OSD OSD Ganesha NFSGW MGR Manila

Push config Start grace period Metadata IO Data IO Get/Put Client State (in RADOS) Get Share/Config + Advertise to ServiceMap Spawn Container in NW Share

/usr/bin /ceph

REST API: Get/Put Shares (Publish Intent) Share: CephFS Name Export Paths Network Share (e.g. Neutron ID+CIDR) Share Server Count

Rook/Kubernetes + Kuryr (net driver)

HA managed by Kubernetes Scale-out & shares managed by mgr

Managing Scale-Out: “CephFSShareNamespace”

10

StatefulSet #1 Pod #1 Pod #2

Ganesha NFSGW Ganesha NFSGW

Service

NFS Client NFS Client

One IP to access NFS Cluster for tenant; only via tenant network # of pods equal to scale-out for “CephFSShareNames pace”; dynamically grow/shrink?? Stable network identifiers for pods!

Openstack Sydney: 2017 November 06

Patrick Donnelly pdonnell@redhat.com Thanks to the CephFS team: John Spray, Greg Farnum, Zheng Yan, Ramana Raja, Doug Fuller, Jeff Layton, and Brett Niver. Homepage: http://ceph.com/ Mailing lists/IRC: http://ceph.com/IRC/

Thanks!

11