OpenStack Swift as a Backend for Git Chmouel Boudjnah - @chmouel - - PowerPoint PPT Presentation

Chmouel Boudjnah � OpenStack Summit 2014 Atlanta OpenStack Swift as a Backend for Git Chmouel Boudjnah - @chmouel - eNovance

The mandatory about me Chmouel Boudjnah - @chmouel - eNovance

About me I am not the author, Fabien Boucher aka: @bzho Chmouel Boudjnah - @chmouel - eNovance Chmouel Boudjnah - @chmouel - eNovance

About me We both work for eNovance Chmouel Boudjnah - @chmouel - eNovance

About me OpenStacker for quite some time now Chmouel Boudjnah - @chmouel - eNovance

So what is Swift? Chmouel Boudjnah - @chmouel - eNovance

What is Swift Object Storage • Not a file system • Eventually Consistent • HTTP REST API • Very Scalable • Software defined Storage • Chmouel Boudjnah - @chmouel - eNovance

Why would you want to use GIT with Swift? Chmouel Boudjnah - @chmouel - eNovance

Why Swift with GIT Why not? Chmouel Boudjnah - @chmouel - eNovance

Why Swift with GIT When you use a file system you have to : Perform backups of your repositories regularly Perform complex and risky operations to extend the storage capabilities for you repositories. You have mostly have to manage it yourself It’s an another piece in your infrastructure You have to manage the downtime Not fun Chmouel Boudjnah - @chmouel - eNovance

So what are the benefits? Chmouel Boudjnah - @chmouel - eNovance

Why Swift with GIT The advantages of using Swift Safety: The inherent replication of a data store. Capability: easily extend storage capacity. High Availability : no single point of failure. Maintainability: allow operations teams to maintain the cluster without any downtime. In an OpenStack environment it’s usually already there . Chmouel Boudjnah - @chmouel - eNovance

Let’s see how git internal works Chmouel Boudjnah - @chmouel - eNovance

Chmouel Boudjnah - @chmouel - eNovance

Chmouel Boudjnah - @chmouel - eNovance

So how did we plug that to Swift? Chmouel Boudjnah - @chmouel - eNovance

We did not want to modify the GIT client binaries. Chmouel Boudjnah - @chmouel - eNovance

We used a pure python GIT library called Dulwich Chmouel Boudjnah - @chmouel - eNovance

Dulwich Create, read, manage loose objects (blob, tree, • commit, tag). Create, read, manage pack files , pack indexes . • Implement the Git smart protocol through git- • upload-pack and git-receive-pack . Implement the Git, HTTP, SSH listeners to start • Dulwich as a Git server. Provide a well designed interface to the storage • backend. (Repo, MemoryRepo). Chmouel Boudjnah - @chmouel - eNovance

How do we use Dulwhich We implemented a custom repo called SwiftRepo as a backend to the git listener of Dulwich which communicate to OpenStack Swift Chmouel Boudjnah - @chmouel - eNovance

So about that SwiftRepo backend how does it work? Chmouel Boudjnah - @chmouel - eNovance

SwiftRepos backend to Dulwich It doesn’t use the standard way to store Git references (one file by reference). It doesn’t store loose objects but instead uses the pack format. In addition to the .index that relate to each pack file we added .info object It makes the Request to Swift concurrently. Chmouel Boudjnah - @chmouel - eNovance

How’s the benchmark then? Chmouel Boudjnah - @chmouel - eNovance

Benchmark Setup Benchmark performed on two architectures : Github : ● Git client from eNovance network <-> GitHub � Dulwich/Swift : ● · Git client from eNovance network <-> Dulwich (Rax compute) <-> Rax Cloudfiles � Three different repos : � ● · Swiftsync (873 objects with 293 commits) ● · Edeploy (2246 objects with 618 commits) ● · Swift (23480 objects with 3212 commit) Chmouel Boudjnah - @chmouel - eNovance

Full repository push Chmouel Boudjnah - @chmouel - eNovance

Clone Chmouel Boudjnah - @chmouel - eNovance

That’s it, hope you enjoyed. Chmouel Boudjnah - @chmouel - eNovance

Eventual Consistentie Why Eventual Consistentie window is not an issue Chmouel Boudjnah - @chmouel - eNovance