Our My first DDoS attack Velocity Europe 2011 Berlin Cosimo - - PowerPoint PPT Presentation

Our My first DDoS attack

Velocity Europe 2011 – Berlin

Cosimo Streppone Operations Lead

<video of Mr. Wolf going to Jimmy's house in Pulp Fiction> this couldn't fit in the PDF... sorry.

http://www.youtube.com/watch?v=hsKv5d0sIlU

my.opera.com/Ao-Trang-Oi/blog/

nginx – secret sauces?

# Pavel's secret gzip tuning sauce gzip on; gzip_disable msie6; gzip_min_length 1100; gzip_buffers 16 8k; gzip_comp_level 3; gzip_types text/plain application/xml application/x-javascript text/css;

nginx – secret sauces?

# Michael's secret file cache sauce

• pen_file_cache max=1000 inactive=20s;
• pen_file_cache_valid 30s;
• pen_file_cache_min_uses 2;
• pen_file_cache_errors on;

nginx – antidos.conf

# More on https://calomel.org/nginx.html client_header_timeout 5; client_body_timeout 10; ignore_invalid_headers on; send_timeout 10; # To limit slowloris-like attacks client_header_buffer_size 4k; large_client_header_buffers 4 4k;

# Cut abusive established connections, # forcing clients to reconnect location ~ ^/Ao-Trang-Oi/blog/ { return 444; }

nginx – drop client connections

nginx backends varnish

nginx – varnish caching

iptraf

GET /Ao-Trang-Oi/blog/show.dml/14715682 HTTP/1.1 User-Agent: 1.{RND 10}.{RND 10} Referrer: http://my.opera.com/Ao-Trang-Oi/ Cache-Control: no-cache Cookie: __utma=218314117.745395330 […] __utmz=218314117.1286774593. […] utmcsr=google|utmccn= […] utmctr=cach%20de%20hoc%20mon […] <... random high speed junk follows ...>

tcpdump of anomalous traffic

GET /Ao-Trang-Oi/blog/?startidx=1295 HTTP/1.1 User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;) Gecko/20030624 Netscape/7.1 (ax) Accept: Accept=text/html,application/xhtml+xml,... Accept-Language: Accept-Language=en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: Accept-Charset=ISO-8859-1,... Referer: http://my.opera.com/Ao-Trang-Oi/blog/ Pragma: no-cache Keep-Alive: 300 ua-cpu: x86 Connection: close

tcpdump of anomalous traffic

cosimo: we're seeing a pretty "interesting" problem within our nginx fronts cosimo: there's a few hosts sending a legitimate HTTP GET request cosimo: followed by a binary stream of random bytes that never ends cosimo: this is just 1 request going on and on cosimo: is there some way to alter the nginx config to shut down these client connections? cosimo: the client is sending something like: cosimo: GET /blah HTTP/1.1 cosimo: Host: ... cosimo: Etc: etc... cosimo: and then random bullshit vr: :) vr: this is nkiller2 vr: haproxy can fight this vr: you can set a timeout http-request vr: don't know if nginx can do this cosimo: cool

OMGWTFBBQ!!!!11111 “this is nkiller2”

#nginx, 14th October 2010

BLAH BLAH BLAH BLAH BLAH BL BLAH BLAH BLAH

PHRACK#66

tcp window zero?

iptables -A -m u32

• -u32 “6&0xFF=0x6 &&

4&0x1FFF=0 && 0>>22&0x3C () 12&0xFFFF=0x0000”

• j ZERO_WINDOW_RECENT

u32 zero window filter

6 & 0xFF = 0x6

4 & 0x1FFF = 0x0 u32 zero window filter

u32 zero window filter

0>>22 & 0x3C () 12 & 0xFFFF = 0x0

0>>22 & 0x3C () 12 & 0xFFFF = 0x0

??

u32 zero window filter

0>>22&0...@12&0xFFFF=0x0000

0>>22&0x3C@12&0xFFFF=0x0000

0>>22& [EMAIL PROTECTED] &0xFFFF=0x0000

0>>22&0x3C@12&0xFFFF=0x0000

0>>22 & 0x3C @ 12 & 0xFFFF = 0x0

u32 zero window filter

iptables rules - logging

$ipt -N ZERO_WINDOW_RECENT $ipt -A INPUT -m u32

• -u32 "6&0xFF=0x6 &&

4&0x1FFF=0 && 0>>22&0x3C@12&0xFFFF=0x0000"

• j ZERO_WINDOW_RECENT

$ipt -A ZERO_WINDOW_RECENT -m recent --set --name ZERO_WINDOW $ipt -A ZERO_WINDOW_RECENT -m recent --update

• -seconds 60 --hitcount 20
• -name ZERO_WINDOW -j LOG
• -log-level info --log-prefix "ZeroWindow"

~18k distinct IPs

iptables rules - blocking

$ipt -N ZERO_WINDOW_RECENT $ipt -A INPUT -m u32

• -u32 "6&0xFF=0x6 &&

4&0x1FFF=0 && 0>>22&0x3C@12&0xFFFF=0x0000"

• j ZERO_WINDOW_RECENT

$ipt -A ZERO_WINDOW_RECENT -m recent –set

• -name ZERO_WINDOW

$ipt -A ZERO_WINDOW_RECENT -m recent –update

• -seconds 60 --hitcount 20
• -name ZERO_WINDOW -j DROP

nginx backends varnish

shields-up.vcl

non-cacheable content cacheable content

nginx backends varnish HTTPS-only traffic

shields-up.vcl

all HTTP content

nginx feels better

10s 20s 0s

Pingdom response time

End 29-Oct-2010

Start 13-Oct-2010 End 29-Oct-2010

Packets/s seen by firewall

¿Questions?

What can we, as Ops, do better?

• Embrace failures and learn from them
• Be fast (no panic/blame, think Mr. Wolf)
• Coordinate (#ops, war rooms, ...)
• Take notes
• Learn TCP/IP
• Know your tools

(tcpdump, tcpflow, strace, nc, iptraf, …)

my base_packages puppet module

class base_packages { $packagelist = [ "ack-grep", "colordiff", "curl", "facter", "git-core", "htop", "iftop", "iptraf", "jed", "joe", "libwww-perl", "logrotate", "lsof", "make", "mc", "oprofile", "psmisc", "rsync", "screen", "svn", "sysstat", "tcpdump", "tcpflow", "telnet", "unzip", "vim", "zip" ] package { $packagelist: ensure => "installed", } }

Thanks to...

• ithilgore (sock-raw.org) for writing nkiller2
• @vr in #nginx for pointing us at nkiller2
• David Falloon for his great “untested” idea
• marc.info for correctly handling “@” in ml
• SANS Institute for the TCP/IP references
• My team at Opera

Danke!