SLIDE 1
FrontendFS
Creating a userspace filesystem in node.js
Clay Smith, New Relic
A file system (or filesystem) is a way
- f storing all data on a data storage
device.
BUILDING A FILESYSTEM IN NODE.JS
FrontendFS Creating a userspace filesystem in node.js Clay Smith, - - PowerPoint PPT Presentation
FrontendFS Creating a userspace filesystem in node.js Clay Smith, - - PowerPoint PPT Presentation
FrontendFS Creating a userspace filesystem in node.js Clay Smith, New Relic BUILDING A FILESYSTEM IN NODE.JS A file system (or filesystem) is a way of storing all data on a data storage device. Simple English Wikipedia BUILDING A FILESYSTEM
SLIDE 2
SLIDE 3
BUILDING A FILESYSTEM IN NODE.JS
Kernel Virtual ¡File ¡System EXT3 Hardware NFS
userspace kernel
a ¡process
SLIDE 4
BUILDING A FILESYSTEM IN NODE.JS
Kernel Virtual ¡File ¡System EXT3 Hardware NFS FUSE
userspace kernel
a ¡process fuse ¡client
SLIDE 5
BUILDING A FILESYSTEM IN NODE.JS
Kernel Virtual ¡File ¡System EXT3 Hardware NFS a ¡process FUSE fuse ¡client
userspace kernel
libfuse
SLIDE 6
BUILDING A FILESYSTEM IN NODE.JS
fuse ¡client.js
good news: node is great at I/O
SLIDE 7
BUILDING A FILESYSTEM IN NODE.JS
How to write a (virtual) filesystem in node.js
CC ¡A ¡SA ¡https://www.flickr.com/photos/ewedistrict/25323526
SLIDE 8
BUILDING A FILESYSTEM IN NODE.JS
- Use the fuse-bindings github project
- https://github.com/mafintosh/fuse-bindings
- Implement required filesystem operations
- Run your node.js code and mount on an
existing directory.
- Success?!?!
SLIDE 9
BUILDING A FILESYSTEM IN NODE.JS
read(filePath, fd, buf, len, pos, cb) { } write(filePath, fd, buf, len, pos, cb) { } create(filePath, mode, cb) { }
- pen(filePath, flags, cb) { }
readdir(filePath, cb) { }
Good ¡list ¡for ¡reference: ¡https://www.cs.hmc.edu/~geoff/classes/hmc.cs135.201001/homework/fuse/fuse_doc.html#function-‑ purposes ¡
getattr(filePath, cb) { }
SLIDE 10
BUILDING A FILESYSTEM IN NODE.JS
getattr(/) getattr(/cat.jpg) create(/cat.jpg, 33188) getattr(/cat.jpg) write(/cat.jpg, 0, len: 65536, buf.length: 65536, pos: 0) write(/cat.jpg, 0, len: 65536, buf.length: 65536, pos: 65536) // ... getattr(/cat.jpg) release(/cat.jpg, 0)
Example: What does a file copy look like?
SLIDE 11
BUILDING A FILESYSTEM IN NODE.JS
In-memory virtual filesystem backed by JS array (yikes)
SLIDE 12
BUILDING A FILESYSTEM IN NODE.JS
var files = [ { name: 'readme.md', data: new Buffer('# test file\n'), } ];
A 'file' is just an item in this array: Buffer()-backed
SLIDE 13
BUILDING A FILESYSTEM IN NODE.JS
readdir() { // ... for (var i = 0; i < files.length; i++) { var buffer = files[i].data; var hash = hasha(buffer).substring(0, 12); files[i].name = `${hash}-${files[i].name}`; files[i].hasFingerprint = true; } }
Let's rename the filename to be a hash!
SLIDE 14
BUILDING A FILESYSTEM IN NODE.JS
releaseFile(filePath, fd) { // Buffer is a WritableStream() var buffer = this.files[fileIndex].data.getContents(); zlib.gzip(buffer, function(error, result) { // create file if compression succeeded. }); }
Like gzip?
SLIDE 15
BUILDING A FILESYSTEM IN NODE.JS
Performance profiling
CC ¡A ¡SA ¡www.flickr.com/photos/libertinus/9231731038 ¡
SLIDE 16
Benchmarking disk or file system IO performance can be tricky at best.
BUILDING A FILESYSTEM IN NODE.JS
James Coyle
SLIDE 17
BUILDING A FILESYSTEM IN NODE.JS
dd: "The Easy Route"
- time sh -c "dd if=/dev/zero of=./in-
memory/tst.tmp bs=4k count=1000 && sync"
- 409600 bytes written 1000 times...
- Result: 3 mb/s (node, FuSE fs)
777827192
SLIDE 18
BUILDING A FILESYSTEM IN NODE.JS
tl;dr - use for fun dev tools, exporting APIs, system learnings
SLIDE 19
BUILDING A FILESYSTEM IN NODE.JS
see also...
- https://github.com/Munter/fusile
- ^ the "real" frontendFS
- https://www.npmjs.com/package/mount-url
- github.com/mafintosh/torrent-mount
SLIDE 20
BUILDING A FILESYSTEM IN NODE.JS
thanks!
@smithclay require('afterparty') - tomorrow at New Relic PDX. https://bit.ly/nodeparty