FileSender Terabyte Challenge Ren e Klomp Edwin Schaap System and - PowerPoint PPT Presentation

FileSender Terabyte Challenge Ren´ e Klomp Edwin Schaap System and Network Engineering University of Amsterdam February 6, 2013 Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 1 / 24

What is FileSender? Meant to be easy for user and sysadmin No extra software requirements Default setup reaches only 150Mb/s Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 2 / 24

Research Question Can we identify bottlenecks in the FileSender application and how can we improve the transfer speeds by reducing or removing these bottlenecks. Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 3 / 24

Environment Tested and excluded the hardware as the bottleneck (except HDD) Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 4 / 24

The bottleneck 0.0 0.2 0.4 0.6 0.8 1.0 1.2 No delay time (s) 0.0 0.2 0.4 0.6 0.8 1.0 1.2 50ms delay time (s) 0.0 0.2 0.4 0.6 0.8 1.0 1.2 100ms delay time (s) 0.0 0.2 0.4 0.6 0.8 1.0 1.2 150ms delay time (s) 0.0 0.2 0.4 0.6 0.8 1.0 1.2 200ms delay time (s) Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 5 / 24

Client side implementation Remove Gaps JavaScript webworkers Parallelisation Concurrent TCP streams (like GridFTP) Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 6 / 24

Server side implementation Handle out-of-order chunks X-Start-Byte HTTP header File based queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 7 / 24

Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 8 / 24

Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 9 / 24

Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 10 / 24

Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 11 / 24

Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 12 / 24

Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 13 / 24

Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 14 / 24

Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 15 / 24

Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 16 / 24

Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 17 / 24

Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 18 / 24

Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 19 / 24

Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 20 / 24

Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 21 / 24

SSL performance 6000 5000 4000 Mb/s Depends on cipher 3000 AES-NI 2000 1000 0 AES128 AES256 RC4 3DES CAMELLIA128 CAMELLIA256 Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 22 / 24

Terabyte Challenge Transferred in 3 hours and 19 minutes (@702Mb/s) Harddisk as bottleneck Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 23 / 24

Upload completed... Questions? made possible by Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 24 / 24