Data Transfers in the Grid: Data Transfers in the Grid: Workload - - PowerPoint PPT Presentation

▶

Apr 05, 2023 674 likes •968 views

Data Transfers in the Grid: Data Transfers in the Grid: Workload Analysis of Globus Globus GridFTP Workload Analysis of GridFTP Nicolas Kourtellis, Lydia Prieto, Gustavo Zarrate, Adriana Iamnitchi Adriana Iamnitchi Nicolas Kourtellis, Lydia

SLIDE 1

Data Transfers in the Grid: Data Transfers in the Grid: Workload Analysis of Workload Analysis of Globus Globus GridFTP GridFTP

Nicolas Kourtellis, Lydia Prieto, Gustavo Zarrate, Nicolas Kourtellis, Lydia Prieto, Gustavo Zarrate, Adriana Iamnitchi Adriana Iamnitchi

University of South Florida University of South Florida

Dan Fraser Dan Fraser

Argonne National Laboratory Argonne National Laboratory

SLIDE 2

2 2

SLIDE 3

3 3

Objective 1 Objective 1: : Quantify volume of transfers Quantify volume of transfers

What is the transfer size distribution? What is the transfer size distribution?

What is the volume of activity for the most active hosts? What is the volume of activity for the most active hosts?

Objective 2 Objective 2: : Understand how tuning capabilities are Understand how tuning capabilities are used used

What are the buffer sizes used during the transfers? What are the buffer sizes used during the transfers? What is the average bandwidth? What is the average bandwidth? What is the utilization of functionalities like streams and What is the utilization of functionalities like streams and stripes? stripes?

Objective 3 Objective 3: : Quantify user base and predict usage Quantify user base and predict usage trends trends

How does the user base evolve over time? How does the user base evolve over time? What are the geographical characteristics of the What are the geographical characteristics of the GridFTP GridFTP data transfers? data transfers?

SLIDE 4

4 4

Outline Outline

Metrics dataset Metrics dataset Surprises and Surprises and … … … … zoom in ( zoom in (TeraGrid TeraGrid) ) Lessons and discussions Lessons and discussions

SLIDE 5

5 5

GridFTP GridFTP Metrics Dataset Metrics Dataset

Field Range of Values Comment Source hostname/host IP String/IPnet Anonymized Start time of the transfer Timestamp Accuracy: ms End time of the transfer Timestamp Accuracy: ms TCP Buffer Size Integer (Bytes) ≥0 Total Number of Bytes Integer (Bytes) ≥0 Number of Streams Integer ≥1 Number of Stripes Integer ≥1 Store or Retrieve Integer (0, 1,2) STOR, RETR, LIST

SLIDE 6

6 6

Started with ~137.5 million records (Jul Started with ~137.5 million records (Jul’ ’05 05 -

Mar’ ’07) 07) Cleaning: Cleaning:

– – transfer size transfer size ≤ ≤0: 0: ~22.8 million records ~22.8 million records – – buffer size <0: buffer size <0: ~1000 records ~1000 records – – directory listings: directory listings: ~3.9 million records ~3.9 million records – – invalid hostnames (e.g., /[B@89712e): invalid hostnames (e.g., /[B@89712e): ~4,600 records ~4,600 records – – ANL ANL-

TeraGrid

TeraGrid testing: testing: ~11.4 million records ~11.4 million records – – duplicate reports: duplicate reports: ~16.8 million records ~16.8 million records – – self transfers (source=destination): self transfers (source=destination): ~5.75 million records ~5.75 million records

Clean database: ~77.2 million records (~56.2%) Clean database: ~77.2 million records (~56.2%)

Metrics Dataset Metrics Dataset

SLIDE 7

7 7

Surprise #1: Transfer Size Distribution Surprise #1: Transfer Size Distribution

5 10 2 4 8 16 32 64 128 256 512 1 2 4 8 16 32 64 128 256 512 1 2 4 8 16 32 64 128 256 512 1

Bytes KB MB Transfer size

% of total

Objective 1 Objective 1: : Quantify volume of transfers Quantify volume of transfers

SLIDE 8

8 8

Zoom Zoom-

in: TeraGrid TeraGrid

Are these results representative for Are these results representative for production grids? production grids?

– – GridFTP GridFTP testing for deployment and learning testing for deployment and learning

Identify transfers from Identify transfers from TeraGRid TeraGRid and and analyze dataset. analyze dataset.

SLIDE 9

9 9

Transfer Size Distribution (TG) Transfer Size Distribution (TG)

5 10 15 20 25 2 4 8 16 32 64 128 256 512 1 2 4 8 16 32 64 128 256 512 1 2 4 8 16 32 64 128 256 512 1

Bytes KΒ MΒ Transfer Size

% of total

Objective 1 Objective 1: : Quantify volume of transfers Quantify volume of transfers

SLIDE 10

10 10

5 10 2 4 8 16 32 64 128 256 512 1 2 4 8 16 32 64 128 256 512 1 2 4 8 16 32 64 128 256 512 1

Bytes KB MB Transfer size

% of total

5 10 15 20 25 2 4 8 16 32 64 128 256 512 1 2 4 8 16 32 64 128 256 512 1 2 4 8 16 32 64 128 256 512 1

Bytes KΒ MΒ Transfer Size

% of total

All TG

SLIDE 11

11 11

Why So Small Transfers? Why So Small Transfers?

There are still many old versions (i.e., before There are still many old versions (i.e., before v3.9.5) of v3.9.5) of GridFTP GridFTP in use. These versions do in use. These versions do not include trace reporting capabilities. not include trace reporting capabilities. Other data transfer protocols and Other data transfer protocols and implementations are used implementations are used Users have turned off the reporting capability Users have turned off the reporting capability Some of the logs are inevitably lost due to the Some of the logs are inevitably lost due to the UDP UDP-

based reporting mechanism

based reporting mechanism The low transfer volumes could suggest a shift The low transfer volumes could suggest a shift towards data towards data-

aware job scheduling (?)

aware job scheduling (?)

SLIDE 12

12 12

Server to Server Transfers Server to Server Transfers

– – High reporting of Self Transfers (more than 1/3) High reporting of Self Transfers (more than 1/3)

21.7% 39.5% 38.8% 72.2% 19.7% 8.2% 0% 10% 20% 30% 40% 50% 60% 70% 80%

InterDomain InterIP SelfTransfers # Transfers Volume

Objective 1 Objective 1: : Quantify volume of transfers Quantify volume of transfers

SLIDE 13

13 13

Top 6 Active Hosts (all) Top 6 Active Hosts (all)

Top 6 hosts traffic adds up to ~28% of total volume Top 6 hosts traffic adds up to ~28% of total volume Next 48 hosts ( Next 48 hosts (IPs IPs) transferred 10s of TB ) transferred 10s of TB

50 100 150 200 250

1 2 3 4 5 6

Host

Volume Transferred (TB)

1.0E+00 1.0E+01 1.0E+02 1.0E+03 1.0E+04 1.0E+05 1.0E+06 1.0E+07 1.0E+08

Number of Transfers

Volume Transferred Number of Transfers

Objective 1 Objective 1: : Quantify volume of transfers Quantify volume of transfers

SLIDE 14

14 14

Number of Transfers & Volume (TG) Number of Transfers & Volume (TG)

500 1000 1500 2000 2500 3000 Aug-05 Sep-05 Oct-05 Nov-05 Dec-05 Jan-06 Feb-06 Mar-06 Apr-06 May-06 Jun-06 Jul-06 Aug-06 Sep-06 Oct-06 Nov-06 Dec-06 Jan-07 Feb-07 Mar-07 Month-Year Number of Transfers (thousands) 10 20 30 40 50 60 70 80

Volum e (TB)

Number of Transfers per Month Total Volume per Month

Objective 1 Objective 1: : Quantify volume of transfers Quantify volume of transfers

SLIDE 15

15 15

Average Transfer Size Average Transfer Size & Total Volume (TG) & Total Volume (TG)

20 40 60 80 100 120 140 160 1 2 3 4 5 6 7 8

TERAGRID SITES

Total Volume (TB)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

Average Transfer Size (GB)

Total Volume (TB) Average Transfer Size (GB)

Objective 1 Objective 1: : Quantify volume of transfers Quantify volume of transfers

SLIDE 16

16 16

Daily Workload (TG) Daily Workload (TG)

Average volume transferred per day: ~ 0.6TB Average volume transferred per day: ~ 0.6TB GridFTP GridFTP doesn doesn’ ’t get weekends free! t get weekends free!

0.00 0.20 0.40 0.60 0.80 1.00 1.20 1.40 Monday TuesdayWednesdayThursday Friday Saturday Sunday

Volume (TB)

5000 10000 15000 20000 25000 30000 35000 40000 45000 50000

Number of Transfers Average Volume per Day Average Number of Transfers per Day

Objective 1 Objective 1: : Quantify volume of transfers Quantify volume of transfers

SLIDE 17

17 17

Monthly Workload (TG) Monthly Workload (TG)

~50,000 transfers per day ~50,000 transfers per day ~1TB per day of total volume ~1TB per day of total volume Lowest around 0.5TB per day Lowest around 0.5TB per day Peaks due to particular days Peaks due to particular days

0.0 0.5 1.0 1.5 2.0 2.5 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 Day of the month Volume (TB) 20000 40000 60000 80000 100000 120000 140000 Number of Transfers

Average Volume Transferred per day Average Number of Transfers per day

Objective 1 Objective 1: : Quantify volume of transfers Quantify volume of transfers

SLIDE 18

18 18

Objective 1: Quantify volume of transfers Objective 1: Quantify volume of transfers

What is the transfer size distribution? What is the transfer size distribution?

What is the volume of activity for the most active hosts? What is the volume of activity for the most active hosts?

Objective 2 Objective 2: : Understand how tuning capabilities are Understand how tuning capabilities are used used

What are the buffer sizes used during the transfers? What are the buffer sizes used during the transfers? What is the average bandwidth? What is the average bandwidth? What is the utilization of functionalities like streams and What is the utilization of functionalities like streams and stripes? stripes?

Objective 3 Objective 3: : Quantify user base and predict usage Quantify user base and predict usage trends trends

How does the user base evolve over time? How does the user base evolve over time? What are the geographical characteristics of the What are the geographical characteristics of the GridFTP GridFTP data transfers? data transfers?

SLIDE 19

19 19

Surprise #2: Usage of Streams and Stripes Surprise #2: Usage of Streams and Stripes

Unreliably reported (from Unreliably reported (from Globus Globus team). team). Reliable observations: Reliable observations:

At least ~20% of the transfers used 4 streams

At least ~20% of the transfers used 4 streams (suggested number by (suggested number by ANL ANL’ ’s s website) website)

At least ~10% of the transfers used a different

At least ~10% of the transfers used a different value, larger than one. value, larger than one.

Maximum number of streams reported: 1010 (!!)

Maximum number of streams reported: 1010 (!!)

Objective 2 Objective 2: : Understand how tuning capabilities are used Understand how tuning capabilities are used

SLIDE 20

20 20

Buffer Size Distribution Buffer Size Distribution

60% from the original table

60% from the original table: OS : OS-

controlled (0 bytes)

controlled (0 bytes)

Most commonly used: 16

Most commonly used: 16– –128KB 128KB

Largest buffer size: 1

Largest buffer size: 1-

2GB (92 records)

2GB (92 records)

1 2 3 4 5 6 7 8 9 4 8 16 32 64 128 256 512 1 2 4 8 16

KB MB Buffer Size % of total

Objective 2 Objective 2: : Understand how tuning capabilities are used Understand how tuning capabilities are used

SLIDE 21

21 21

Average Bandwidth Distribution Average Bandwidth Distribution

Peak: 128

Peak: 128-

256Mbps, ~7.7 million records

256Mbps, ~7.7 million records

Most common (58%): 4Mbps

Most common (58%): 4Mbps— —1Gbps 1Gbps

2 4 6 8 10 2 4 8 16 32 64 128 256 512 1 2 4 8 16 32 64 128 256 512 1 2 4 8 16 32 64 128 256 512 1

bps Kbps Mbps Average Bandwidth % of total

SLIDE 22

22 22

Average Bandwidth Distribution (TG) Average Bandwidth Distribution (TG)

Compared to the total dataset: Compared to the total dataset:

=> => The region of 4Mbps to 1Gbps includes more than 85% of the The region of 4Mbps to 1Gbps includes more than 85% of the transfers (58% for the whole dataset) transfers (58% for the whole dataset)

2 4 6 8 10 12 14 16 18 20 1 2 4 8 16 32 64 128 256 512 1 2 4 8 16 32 64 128 256 512 1 2 4

Kbps Mbps Gbps Average Bandwidth

% of total

SLIDE 23

23 23

Objective 1: Quantify volume of transfers Objective 1: Quantify volume of transfers

What is the transfer size distribution? What is the transfer size distribution?

What is the volume of activity for the most active hosts? What is the volume of activity for the most active hosts?

Objective 2: Understand how tuning capabilities are Objective 2: Understand how tuning capabilities are used used

What are the buffer sizes used during the transfers? What are the buffer sizes used during the transfers? What is the average bandwidth? What is the average bandwidth? What is the utilization of functionalities like streams and What is the utilization of functionalities like streams and stripes? stripes?

Objective 3 Objective 3: : Quantify user base and predict usage Quantify user base and predict usage trends trends

How does the user base evolve over time? How does the user base evolve over time? What are the geographical characteristics of the What are the geographical characteristics of the GridFTP GridFTP data transfers? data transfers?

SLIDE 24

24 24

Surprise #3: Geographic Distribution Surprise #3: Geographic Distribution

USA: 78.4% or ~50.8 million transfers and 82.9% or ~1.7 PB USA: 78.4% or ~50.8 million transfers and 82.9% or ~1.7 PB Canada+Taiwan+Japan+Spain:~14M transfers and 346TB Canada+Taiwan+Japan+Spain:~14M transfers and 346TB 49 different countries and 446 different cities (178 cities from 49 different countries and 446 different cities (178 cities from USA) USA)

0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% CA TW JP IT DE AU CH ES GB AT

# transfers volume

Objective 3 Objective 3: : Quantify user base and predict usage trends Quantify user base and predict usage trends

SLIDE 25

25 25

User and Domain Evolution (all) User and Domain Evolution (all)

Continuing increase of user and organization population Continuing increase of user and organization population Forecasts: 67 new Forecasts: 67 new IPs IPs and 14 new domains per month and 14 new domains per month

y = 14.132x + 67.078 R2 = 0.9107

200 400 600 800 1000 1200 Jul-05 Aug-05 Sep-05 O ct-05 Nov-05 Dec-05 Jan-06 Feb-06 M ar-06 Apr-06 M ay-06 Jun-06 Jul-06 Aug-06 Sep-06 O ct-06 Nov-06 Dec-06 Jan-07 Feb-07 M ar-07

IPs

50 100 150 200 250 300 350 400

Dom ains

# IPs # Domains Linear Fit (# Domains)

Objective 3 Objective 3: : Quantify user base and predict usage trends Quantify user base and predict usage trends

SLIDE 26

26 26

Summary of Results Summary of Results

Many transfers in the range of Many transfers in the range of KBs KBs to 10s MB to 10s MB (peak in 16MB (peak in 16MB-

32MB).

32MB).

– – relevant for setting up realistic simulations. relevant for setting up realistic simulations. – – previous work assume different, larger file sizes. previous work assume different, larger file sizes.

Bandwidth measured in previous work is Bandwidth measured in previous work is confirmed by our workload analysis. confirmed by our workload analysis. Tuning parameters: Tuning parameters:

– – Users tend not to set the buffer size explicitly (60%), leaving Users tend not to set the buffer size explicitly (60%), leaving it to it to the OS the OS – – The unexpectedly small transfers do not justify tuning The unexpectedly small transfers do not justify tuning GridFTP GridFTP parameters (stripes and streams) parameters (stripes and streams)

The usage of The usage of Globus Globus GridFTP GridFTP is growing over is growing over time in terms of time in terms of IPs IPs (users), domains (users), domains (organizations), and volume transferred. (organizations), and volume transferred. Missing some of the big players Missing some of the big players

SLIDE 27