Entropy Gradient Reversal Or: How we survived CVE-2008-0166 Kees Cook Matt Zimmerman Opsen 2014-04-22
May 2008
“why do you have my key?”
Fix it quick! openssl (0.9.8g-9) unstable; urgency=high [ Christoph Martin ] * Include updated debconf translations (closes: #473477, #461597, #461880, #462011, #465517, #475439) [ Kurt Roeckx ] * ssleay_rand_add() really needs to call MD_Update() for buf. -- Kurt Roeckx <kurt@roeckx.be> Wed, 07 May 2008 20:32:12 +0200
“Time to release an advisory”
“hold on just a tick…”
Relevant to your interests From: Florian Weimer <fw@deneb.enyo.de> To: vendor-sec@lst.de Subject: [vendor-sec] Broken openssl in Debian Date: Thu, 08 May 2008 01:52:17 +0200 Debian ships a bogus modification of OpenSSL which disables proper seeding of the PRNG (the patch to crypto/rand/md_rand.c). The issue is already semi-public, thanks to an uncoordinated upload by the maintainer. This should only affect Debian-derived distros.
Bank holiday Monday
Meanwhile, downstream...
Cast of characters ● Kees Cook, Ubuntu security team (on vacation in Canada) ● Jamie Strandboge, Ubuntu security team ● Colin Watson, Debian/Ubuntu OpenSSH maintainer ● Matt Zimmerman, Ubuntu CTO ● Florian Weimer, Red Hat Product Security Team ● James Troup, Debian/Canonical
Test case /* gcc -o omfg omfg.c -lcrypto */ #include <openssl/rand.h> #include <stdio.h> int main() { unsigned char data[128]; unsigned i; RAND_bytes(data, 128); for (i = 0; i < 128; ++i) printf("%08X", (unsigned)data[i]); puts(""); } $ for i in $(seq 1 1000000); do ./omfg; done | sort -u | wc -l 32003 <- this number should be 1000000
Sources of “entropy” ● process ID ● CPU endianness Find some alternate endian machines and LD_PRELOAD “getpid”: http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/trusty/openssh- blacklist/trusty/view/head:/generate-blacklist.sh
It gets worse ● All SSH keys generated by this code ● All SSL certificates generated by this code ● All OpenVPN certificates generated by this code
Response plan “This is outstandingly bad. At the very least, we need to: 1- fix it (patch is trivial) 2- educate people that they need to regenerate their keys and certs 3- generate the list of "weak keys" and refuse to use them for SSH 4- hope no one has packet captures from weak SSL connections 5- hope the world doesn't burn 6- find out if places like cacert.org (and others) are using Debian-derived openssl 7- pray we are not brutally lampooned”
Choosing the lesser evil
May 13 2008
Updating your system: 1. Install the security updates Once the update is applied, weak user keys will be automatically rejected where possible (though they cannot be detected in all cases). If you are using such keys for user authentication, they will immediately stop working and will need to be replaced (see step 3). OpenSSH host keys can be automatically regenerated when the OpenSSH security update is applied. The update will prompt for confirmation before taking this step. 2. Update OpenSSH known_hosts files The regeneration of host keys will cause a warning to be displayed when connecting to the system using SSH until the host key is updated in the known_hosts file. The warning will look like this: [...] 3. Check all OpenSSH user keys The safest course of action is to regenerate all OpenSSH user keys, except where it can be established to a high degree of
certainty that the key was generated on an unaffected system. 4. Regenerate any affected user keys Check whether your key is affected by running the ssh-vulnkey OpenSSH keys used for user authentication must be manually tool, included in the security update. By default, ssh-vulnkey regenerated, including those which may have since been will check the standard location for user keys (~/.ssh/id_rsa, transferred to a different system after being generated. ~/.ssh/id_dsa and ~/.ssh/identity), your authorized_keys file New keys can be generated using ssh-keygen, e.g.: (~/.ssh/authorized_keys and ~/.ssh/authorized_keys2), and the $ ssh-keygen system's host keys (/etc/ssh/ssh_host_dsa_key and Generating public/private rsa key pair. /etc/ssh/ssh_host_rsa_key). Enter file in which to save the key (/home/user/.ssh/id_rsa): To check all your own keys , assuming they are in the standard Enter passphrase (empty for no passphrase): locations (~/.ssh/id_rsa, ~/.ssh/id_dsa, or ~/.ssh/identity): Enter same passphrase again: $ ssh-vulnkey Your identification has been saved in /home/user/.ssh/id_rsa. To check all keys on your system : Your public key has been saved in /home/user/.ssh/id_rsa.pub. $ sudo ssh-vulnkey -a The key fingerprint is: To check a key in a non-standard location : 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00 user@host $ ssh-vulnkey /path/to/key 5. Update authorized_keys files (if necessary) If ssh-vulnkey says "Unknown (no blacklist information)", Once the user keys have been regenerated, the relevant public then it has no information about whether that key is affected. keys must be propagated to any authorized_keys files on If in doubt, destroy the key and generate a new one. remote systems. Be sure to delete the affected key.
Details USN-612-3 addressed a weakness in OpenSSL certificate and keys generation in OpenVPN by adding checks for vulnerable certificates and keys to OpenVPN. A regression was introduced in OpenVPN when using TLS, multi-client/server mode, and specifying a user or group which caused OpenVPN to not start when using valid SSL certificates. It was also found that openssl-vulnkey from openssl-blacklist would fail when stderr was not available. This caused OpenVPN to fail to start when used with applications such as NetworkManager. This update fixes these problems. We apologize for the inconvenience.
May 14 2008
How did we get here?
20 months earlier…
How did we get here?
How did we get here? Debian package maintainer, openssl
How did we get here? “If it helps with debugging, I'm in favor of removing them ” - Ulf Möller, OpenSSL contributor “use -DPURIFY” - Geoff Thorpe, Member, OpenSSL development team “So yes I think not using the uninitialized memory [...] helps valgrind.” - Marco Roeland, some dude on the internet
How did we get here? openssl (0.9.8b-1) unstable; urgency=low * libssl0.9.8.postinst: The check to see if something was installed wasn't working. * New upstream release * libssl0.9.8.postinst: Add workaround to find the name of the init - New functions added (EVP_CIPHER_CTX_new, EVP_CIPHER_CTX_free), bump shlibs. script for proftpd and dovecot. - CA.pl/CA.sh now calls openssl ca with -extensions v3_ca, setting CA:TRUE * libssl0.9.8.postinst: Use invoke-rc.d when it's available. instead of FALSE. * Change Standards-Version to 3.7.0: - CA.pl/CA.sh creates crlnumber now. (Closes: #347612) - Make use of invoke-rc.d * Run debconf-updatepo, which really already was in the 0.9.8a-8 version * Add comment to README.Debian that rc5, mdc2 and idea have been as it was uploaded. disabled (since 0.9.6b-3) (Closes: #362754) * Add Galician debconf translation. Patch from * Don't add uninitialised data to the random number generator. This stop Jacobo Tarrio <jtarrio@trasno.net> (Closes: #361266) valgrind from giving error messages in unrelated code. * libssl0.9.8.postinst makes uses of bashisms (local variables) (Closes: #363516) so use #!/bin/bash * Put the FAQ in the openssl docs. * libssl0.9.8.postinst: Call set -e after sourcing the debconf * Add russian debconf translations from Yuriy Talakan <yt@amur.elektra.ru> script. (Closes #367216) * libssl0.9.8.postinst: Change list of service that may need to be restarted: -- Kurt Roeckx <kurt@roeckx.be> Thu, 4 May 2006 20:40:03 +0200 - Replace ssh by openssh-server - Split postgresql in postgresql-7.4 postgresql-8.0 postgresql-8.1 - Add: dovecot-common bind9 ntp-refclock ntp-simple openntpd clamcour fetchmail ftpd-ssl proftpd proftpd-ldap proftpd-mysql proftpd-pgsql
What happened? ● A subtle bug was discovered ● A well-meaning developer attempted to fix it ● They sought a trusted opinion about their patch ○ ...and seemed to receive it ● An even more subtle bug was introduced ○ There was just enough entropy to obscure the issue ● A long time passed! ● A lot of people had a bad day^Wweek^Wlife
It could have been worse ● The issue could have become public before fixes were available ● A Debian advisory was almost released several days earlier, without key blacklists and other critical components ● It could have happened in upstream OpenSSL
Post hoc
Criticism “My issue is that it was committed to a public repository five days before an advisory was issued. Only a single attacker has to notice that and realise its import in order to start exploiting vulnerable systems – and I will be surprised if that has not happened.” - Ben Laurie, Member, OpenSSL core team
Criticism
Recommend
More recommend
