2007 Day In The Life Mistakes We Dont Want To Repeat Duane Wessels - - PowerPoint PPT Presentation

2007 day in the life mistakes we don t want to repeat
SMART_READER_LITE
LIVE PREVIEW

2007 Day In The Life Mistakes We Dont Want To Repeat Duane Wessels - - PowerPoint PPT Presentation

2007 Day In The Life Mistakes We Dont Want To Repeat Duane Wessels The Measurement Factory/CAIDA WIDE+CAIDA Workshop #9 January 20, 2008 WIDE+CAIDA #9 0 The Measurement Factory OARC Disk Space Problem: The OARC server did not have


slide-1
SLIDE 1

2007 Day In The Life Mistakes We Don’t Want To Repeat

Duane Wessels The Measurement Factory/CAIDA WIDE+CAIDA Workshop #9 January 20, 2008

WIDE+CAIDA #9 The Measurement Factory

slide-2
SLIDE 2

OARC Disk Space

Problem:

  • The OARC server did not have enough disk space to store

all the data it should have recieved.

  • FYI: DITL-2007 is about 800 GB, mostly from 4 roots.

Solution:

  • More disk space
  • New file server

WIDE+CAIDA #9 1 The Measurement Factory

slide-3
SLIDE 3

Bandwidth Bottleneck

  • r Local Disk Shortage

Problem:

  • Some collector sites cannot upload pcap files fast enough

and run out of local disk space. Solution:

  • Use an intermediate storage node?
  • Operated by whom?

WIDE+CAIDA #9 2 The Measurement Factory

slide-4
SLIDE 4

Senders Can’t List or Delete

Problem:

  • Once data has been uploaded, senders cannot list, read, or

remove the file on the OARC server. Solution:

  • There was talk of providing access via sftp?

WIDE+CAIDA #9 3 The Measurement Factory

slide-5
SLIDE 5

Partial Uploads

Problem:

  • Partial uploads were hard to detect because the receiver did

not use temporary file names. Solution:

  • This bug has been fixed. Files will be given their final name
  • nly if the upload suceeds.

WIDE+CAIDA #9 4 The Measurement Factory

slide-6
SLIDE 6

No Metadata

Problem:

  • OARC upload interface does not provide a way to also send
  • metadata. Senders must write down or otherwise remember

metadata details until data is indexed in DatCat. Solution:

WIDE+CAIDA #9 5 The Measurement Factory

slide-7
SLIDE 7

Varying pcap Size and Boundaries

Problem:

  • Senders use different time limits on pcap files (1 hour, 5

minutes, variable)

  • File start/stop times based on when the program started.

Solution:

  • Use dnscap with one hour time limit.
  • New file will be started at the top of each hour.

WIDE+CAIDA #9 6 The Measurement Factory

slide-8
SLIDE 8

Clock Skew

Problem:

  • Queries containing timestamps uncovered clock skew issues.

Solution:

  • Send skew-detecting queries to all server unicast addresses?
  • What about load-balanced nodes?
  • Convince operators to make their NTPs readable?

WIDE+CAIDA #9 7 The Measurement Factory

slide-9
SLIDE 9

Truncated Packets

Problem:

  • Some folks forgot ’-s 0’ on tcpdump or used ’-s 1500’.

Solution:

  • Use dnscap instead.

WIDE+CAIDA #9 8 The Measurement Factory

slide-10
SLIDE 10

Extra Data

Problem:

  • Some pcap files contained “extra data” – i.e., packets from

nearby DNS servers using the same infrastructure. Solution:

  • Perhaps we should normalize pcap files after they have been

received.

WIDE+CAIDA #9 9 The Measurement Factory

slide-11
SLIDE 11

VLAN tags

Problem:

  • Some pcap files were captured from VLAN-enabled networks

and create a minor nuisance for pcap readers and bpf filters. Solution:

  • Perhaps we should normalize pcap files after they have been

received.

WIDE+CAIDA #9 10 The Measurement Factory

slide-12
SLIDE 12

The End