Testing TLS Hubert Kario Quality Engineer 24-10-2015 2014 - PowerPoint PPT Presentation

Testing TLS Hubert Kario Quality Engineer 24-10-2015

2014

Heartbleed 24-10-2015 3/55

OpenSSL CCS bug 24-10-2015 4/55

gotofail 24-10-2015 5/55

Certifjcate handling 24-10-2015 6/55

CVE-2014-6321 in schannel a.k.a. Winshock 24-10-2015 7/55

POODLE 24-10-2015 8/55

2015

FREAK 24-10-2015 10/55

LOGJAM 24-10-2015 11/55

State of testing

OSS projects w/test plans <20% No testing No plan Source: Farooq & Quadri, 2011 24-10-2015 13/55

OSS projects w/test tools 40% No testing No tooling Source: Farooq & Quadri, 2011 24-10-2015 14/55

Code coverage tools No testing <50% No coverage Source: Farooq & Quadri, 2011 24-10-2015 15/55

Bad error handling Incorrect 8% 92% Missing Source: Yan, Luo, Zhuang, Rodrigues, et al, 2014 24-10-2015 16/55

Unit tests vs bugs No testing 77% Reproducible in unit testing Source: Yan, Luo, Zhuang, Rodrigues, et al, 2014 24-10-2015 17/55

OSS TLS libraries OpenSSL NSS GnuTLS Framework N° tests 100-200 >7000 100-200 Negative tests 24-10-2015 18/55

Test coverage 120 100 80 Tests/LOC 60 40 20 0 OpenSSL GnuTLS NSS “decent” sqlite 24-10-2015 19/55

Test coverage 12 10 8 Tests/LOC 6 4 2 0 OpenSSL GnuTLS NSS “decent” sqlite 24-10-2015 20/55

Test coverage 1.2 1 0.8 Tests/LOC 0.6 0.4 0.2 0 OpenSSL GnuTLS NSS “decent” sqlite 24-10-2015 21/55

Why is that?

Libraries and bad data 24-10-2015 23/55

Invisible bugs 24-10-2015 24/55

Fuzzy testing 24-10-2015 25/55

Compatibility fears 24-10-2015 26/55

Fears of untested code 24-10-2015 27/55

Fixing the problem

Duplication of effort 24-10-2015 29/55

Full TLS handshake ClientHello --------> ServerHello Certificate ServerKeyExchange CertificateRequest <-------- ServerHelloDone Certificate ClientKeyExchange CertificateVerify ChangeCipherSpec Finished --------> ChangeCipherSpec <-------- Finished ApplicationData < --------> ApplicationData 24-10-2015 30/55

Existing fuzzers 24-10-2015 31/55

TLS testing (and fuzzing) 24-10-2015 32/55

Timing information 24-10-2015 33/55

Tlsfuzzer (and tlslite-ng)

Use cases 1. Manual run (setup) 2. Automated run 24-10-2015 35/55

Architecture generator tlslite-ng fuzzer OpenSSL NSS GnuTLS executor etc. 24-10-2015 36/55

Generator architecture templates scanner generator conversation 24-10-2015 37/55

Fuzzer architecture conversation fuzz verify conversation 24-10-2015 38/55

Runner architecture expect conversation generate command 24-10-2015 39/55

Architecture generator tlslite-ng fuzzer OpenSSL NSS GnuTLS executor etc. 24-10-2015 40/55

Correct run $ openssl s_server -key /tmp/localhost.key -cert /tmp/localhost.crt -www >/dev/null 2>&1 $ PYTHONPATH=. python scripts/test-interleaved-application-data-and- fragmented-handshakes-in-renegotiation.py Application data inside Finished... OK Application data inside Client Key Exchange... OK Application data inside Client Hello... OK Test end successful: 3 failed: 0 24-10-2015 41/55

Failing run $ openssl s_server -key /tmp/localhost.key -cert /tmp/localhost.crt -www >/dev/null 2>&1 $ PYTHONPATH=. python scripts/test-interleaved-application-data-and- fragmented-handshakes-in-renegotiation.py (...snip...) Application data inside Client Hello... Error encountered while processing node <tlsfuzzer.expect.ExpectServerHello object at 0x7f0ac61d3310> with last message being: <tlslite.messages.Message object at 0x7f0ac5f36a50> (...snip...) AssertionError: Unexpected message from peer: Alert(fatal, unexpected_message) Test end successful: 1 failed: 2 24-10-2015 42/55

Example test case conversation = Connect("localhost", 4433) node = conversation ciphers = [CipherSuite.TLS_RSA_WITH_AES_128_CBC_SHA] node = node.add_child(ClientHelloGenerator(ciphers)) node = node.add_child(ExpectServerHello()) node = node.add_child(ExpectCertificate()) node = node.add_child(ExpectServerHelloDone()) node = node.add_child(ClientKeyExchangeGenerator()) node = node.add_child(ChangeCipherSpecGenerator()) node = node.add_child(FinishedGenerator()) node = node.add_child(ExpectChangeCipherSpec()) node = node.add_child(ExpectFinished()) node = node.add_child(ApplicationDataGenerator( bytearray(b"hello server!\n"))) node = node.add_child(AlertGenerator( AlertLevel.warning, AlertDescription.close_notify)) node = node.add_child(ExpectAlert()) node.next_sibling = ExpectClose() 24-10-2015 43/55

Example test case conversation = Connect("localhost", 4433) node = conversation ciphers = [CipherSuite.TLS_RSA_WITH_AES_128_CBC_SHA] node = node.add_child(ClientHelloGenerator(ciphers)) node = node.add_child(ExpectServerHello()) node = node.add_child(ExpectCertificate()) node = node.add_child(ExpectServerHelloDone()) node = node.add_child(ClientKeyExchangeGenerator()) node = node.add_child(ChangeCipherSpecGenerator()) node = node.add_child(FinishedGenerator()) node = node.add_child(ExpectChangeCipherSpec()) node = node.add_child(ExpectFinished()) node = node.add_child(ApplicationDataGenerator( bytearray(b"hello server!\n"))) node = node.add_child(AlertGenerator( AlertLevel.warning, AlertDescription.close_notify)) node = node.add_child(ExpectAlert()) node.next_sibling = ExpectClose() 24-10-2015 44/55

Decision tree Connect ClientHelloGenerator ExpectServerHello AlertGenerator ExpectAlert ExpectClose 24-10-2015 45/55

Invalid extension test case conversation = Connect("localhost", 4433) node = conversation ciphers = [CipherSuite.TLS_RSA_WITH_AES_128_CBC_SHA] ext = {0 : # server_name extension ID lambda _: TLSExtension().create(0, bytearray(b'\xff'*4))} node = node.add_child(ClientHelloGenerator(ciphers, extensions=ext)) node = node.add_child(ExpectAlert(AlertLevel.fatal, AlertDescription.decode_error)) alert_node = node node = node.add_child(ExpectCose()) alert_node.next_sibling = ExpectClose() 24-10-2015 46/55

Handshake message format Byte + 0 Byte + 1 Byte + 3 Byte + 4 Message Bytes 0..4 Message length type Bytes 5..8 Version Random (32 bytes) Session_ID ... Session_ID (0-32 bytes) length 24-10-2015 47/55

Truncated message test case conversation = Connect("localhost", 4433) node = conversation ciphers = [CipherSuite.TLS_RSA_WITH_AES_128_CBC_SHA] node = node.add_child(truncate_handshake( ClientHelloGenerator(ciphers), 1)) node = node.add_child(ExpectAlert(AlertLevel.fatal, AlertDescription.decode_error)) alert_node = node node = node.add_child(ExpectCose()) alert_node.next_sibling = ExpectClose() 24-10-2015 48/55

Padded message test case conversation = Connect("localhost", 4433) node = conversation ciphers = [CipherSuite.TLS_RSA_WITH_AES_128_CBC_SHA] node = node.add_child(pad_handshake(ClientHelloGenerator(ciphers), pad=bytearay(b'\xff\xff')) node = node.add_child(ExpectAlert(AlertLevel.fatal, AlertDescription.decode_error)) alert_node = node node = node.add_child(ExpectCose()) alert_node.next_sibling = ExpectClose() 24-10-2015 49/55

Features ● SSLv3, TLSv1.0, TLSv1.1 and TLSv1.2 ● AES-CBC, AES-GCM, 3DES, RC4 and NULL ciphers ● MD5, SHA1, SHA256 and SHA384 HMAC ● RSA, SRP, SRP_RSA, DHE and DH_anon key exchange ● Encrypt-then-MAC ● TACK certifjcate pinning ● Client certifjcates ● Secure renegotiation ● TLS_FALLBACK_SCSV ● Next Protocol Negotiation ● ChaCha20/Poly1305 (soon™) ● ECDHE (soon™) 24-10-2015 50/55

Missing stuff ● Drafts of TLSv1.3 ● Extended master secret ● PSK key exchange ● ALPN ● AES-CCM ● CAMELLIA (CBC and GCM) ● ECDSA, DSA certifjcates ● Drafts of Curve25519 ● Raw keys, GPG keys ● Heartbeat protocol ● Kerberos 24-10-2015 51/55

Missing stuff ● Test cases! 24-10-2015 52/55

Results 24-10-2015 53/55

Contributing ● https://github.com/tomato42/tlsfuzzer ● https://github.com/tomato42/tlslite-ng ● GPLv2 for tlsfuzzer ● LGPLv2 for tlslite-ng ● Tags review request and help wanted 24-10-2015 54/55

Questions? Contact: hkario@redhat.com Project: https://github.com/tomato42/tlsfuzzer