Fixing a mess: Symlinks and security Opening Windows to a Wider - - PowerPoint PPT Presentation

Opening Windows to a Wider World

Opening Windows to a Wider World

Fixing a mess: Symlinks and security Jeremy Allison Samba Team jra@samba.org

Opening Windows to a Wider World

Opening Windows to a Wider World

• Fileserving is conceptually very simple. Samba (smbserver)

was a simple proxy for client requests.

• Client says “open this pathname”

–Server makes the pathname relative to the exported share

(*).

–Server returns handle internally mapped to file descriptor.

• Client uses handle to make further requests

(read/write/close/etc.).

• Easy – job done !
• (*) The devil is in the details.

Opening Windows to a Wider World

Opening Windows to a Wider World

Pathname problems

• Windows pathnames are different from UNIX/Posix/Linux.
• Prohibited characters ':', different character sets.
• Local pathnames can start with “/”, and contain “.” and “..”
• Client-sent pathnames need processing:

–Early code stripped off any leading “/” –Walked the path removing “.” –Walked the path backing up to the previous “/” on “..” –Ensured the resulting path started with “./”

• Amazingly enough, this protects from all client attempts to

break out of an exported share.

Opening Windows to a Wider World

Opening Windows to a Wider World

• What are symlinks ?

–Usually 4096 byte key/value store. –Key is pathname, value stored in place of file contents. –Metadata in file system causes kernel to treat them

differently.

–Came from Berkeley in BSD 4.2 in 1983.

• When parsing pathnames kernel reads contents of symlink

(value)

–If contents start with “/” replace current path and restart

parsing

–If contents don't start with “/” replace last parsed component

Opening Windows to a Wider World

Opening Windows to a Wider World

Symlink effects

• Samba users started adding symlinks inside shares to make

their lives easier.

–Smbd server didn't treat symlink paths any differently, it just

follows them.

• Less than one minute later someone added an absolute

symlink to /etc/passwd. Oh.

–“Houston, we have a problem...” –We needed to find a way to say “I'm sorry Dave, I can't do

that”.

Opening Windows to a Wider World

Opening Windows to a Wider World

The Unbearable Uselessness of POSIX

• Casting around inside modern POSIX, O_NOFOLLOW

seems like a good idea.

• From the POSIX spec:

–“O_NOFOLLOW - If pathname is a symbolic link, then the

• pen fails, with the error ELOOP.”

–This does not do what you think it does. Read on.

• “Symbolic links in earlier components of the

pathname will still be followed.”

–THIS IS NEVER WHAT YOU WOULD WANT ! –Only way to use this is to walk the path, component by

Opening Windows to a Wider World

Opening Windows to a Wider World

The lstat solution ?

• OK, O_NOFOLLOW is a bust. How about changing all calls

to stat() into lstat().

–lstat() can tell you if a pathname is a symbolic link. –Only works for the last component.

• This suffers from the same problem as O_NOFOLLOW.

–Path must be walked, component by component.

• Unfortunately that's not what smbd does.

–On client request for a share, smbd changes directory to the

root of that share. All calls are then relative to the $cwd.

Opening Windows to a Wider World

Opening Windows to a Wider World

POSIX giveth and POSIX taketh away

• Next possible solution: realpath().

–“realpath() expands all symbolic links and resolves

references to /./, /../ and extra '/' characters in the null- terminated string named by path to produce a canonicalized absolute pathname”.

• Seems perfect. How to use:

–Process client name as previously described. –If required, pass relative name from client to realpath().

• Obtains absolute path guaranteed to have no symlinks.

– Check that the exported share path are the leading

Opening Windows to a Wider World

Opening Windows to a Wider World

The End of the talk !

• Using realpath() provides a perfect solution.

–But what about.. ?

• Can be controlled by a smb.conf switch to allow people who

want insecure symlinks to work.

–No really, what about.. ?

• Allows all smbd pathnames to be $cwd-relative to the root of

the share, no code disruption or major changes.

–All I'm saying is can't the client..

• SHUT UP, SHUT UP, I CAN'T HEAR YOU. LA LA LA !!!

And so things stayed for many years..

Opening Windows to a Wider World

Opening Windows to a Wider World

Solution

• Symlinks can only be created on the exported filesystem by

an authenticated user with local access.

–So even if smbd did allow access outside the share via a

symlink, it's no worse than what a logged-on user can do locally.

• This ignores NFS.

–If the filesystem is exported by NFS as well as smbd, then

an attacker can create a remote symlink that smbd will follow.

• Don't use NFS, don't export the same areas via both

protocols.

–But realpath() still saves us, right ?

Opening Windows to a Wider World

Opening Windows to a Wider World

More Cracks in the Walls

• UNIX extensions for SMB1 get implemented in smbd.

–Little consideration of security concerns when designed or

implemented.

–Allowing SMB1 to give full POSIX semantics enabling SMB1

UNIX clients to host user home directories or even full remote boot was the priority.

• Client can create symlinks on the server.

–In a spectacular failure of judgement, I implement this as a

direct call to symlink() inside smbd. No modification of the incoming symlink value sent from the client.

–“Well it might be useful to have the server-side processes

Opening Windows to a Wider World

Opening Windows to a Wider World

Structural Failure Imminent

• A user pointed out that Samba didn't implement the

Windows “open with backup intent” flag.

–If a connected user has SeBackup Privilege they can ignore

ACL restrictions.

–For smbd, this means doing everything as root.

• Code quality had improved since the early days. Explicitly

doing everything as root needs some careful consideration.

• Parallel filename resolution path created –

check_reduced_name_with_privilege().

–Complex solution outside of main code paths.

Opening Windows to a Wider World

Opening Windows to a Wider World

The House Falls Down

• On Thu, 15 Dec 2016, security@samba.org gets an email

from Jann Horn of the Google Zero Day Initiative with the subject: “security bug report: symlink race permits opening files outside share directory”

• Included was a proof of concept patch to smbclient that

exposes the race condition inherent in checking the client pathname with realpath() followed by the open() call.

–I'm lucky enough to see this. Immediately I know we're

• hosed. I squirm for a few minutes as the proof of concept

never works unless smbd is slowed down via strace().

–We have to fix this properly. This is going to be painful.

Opening Windows to a Wider World

Opening Windows to a Wider World

Fun with TOCTOU

1) Open file “a/long/pathname/secret” Client Server 1) OK, Is a/long/pathname/secret ins realpath(a/long/pathname/secret) → /share/a/long/pathname/secret Looks good to me ! 2) Rename “a/long/pathname” to “a/long/get-out-of-my-way” 2) realpath(a/long/pathname) → /share/a/long/pathname Looks good to me ! Rename completed. 3) Symlink “a/long/pathname” to “/etc” 3) a/long/pathname is good. Anything i 1) open(a/long/pathname/secret) Here Have a nice day !

Opening Windows to a Wider World

Opening Windows to a Wider World

• Immediate patch idea – change UNIX extensions symlink

to add “smbsym:” prefix to any incoming symlink target. Strip this out when client requests symlink target.

• Works by making any symlink target implicitly into a path

relative to the directory including it.

–Nice idea, but breaks lots of existing setups. –Targets including “../” still need careful handling. –Doesn't fix NFS or local races.

• Doesn't work. Not a complete or safe fix. Might be worth

revisiting when creating SMB3 UNIX extensions spec.

Opening Windows to a Wider World

Opening Windows to a Wider World

• Let's do the path walk using O_NOFOLLOW.
• Has become standard practice for this kind of fix:

–fd1 = open(“/”, O_NOFOLLOW); –fd2 = openat(fd1, “a”, O_NOFOLLOW); –fd3 = openat(fd2, “long”, O_NOFOLLOW); –Until pathname resolved.

• Works but is patented. (IBM filed in 2010, granted 2015).
• Means large change to Samba codebase, also to Samba

VFS interface.

–Goal is least intrusive fix for existing Samba versions.

Opening Windows to a Wider World

Opening Windows to a Wider World

Adopted fix

• How to safely process a path:

–Strip off last component “a/long/pathname/secret” →

“a/long/pathname”.

–chdir(“a/long/pathname”) –realpath(“.”) → ensure result is under exported share path. –open(“secret”, O_NOFOLLOW)

• The key is after the chdir(), the realpath(“.”) call is

guaranteed safe against race conditions. Even if another client renames and symlink races the chdir path, once we're already there we're safe.

Opening Windows to a Wider World

Opening Windows to a Wider World

• The previous code breaks symlink targets that point inside

a share path. Many Samba installations depend on this and following them is allowed.

• Solution is to always use O_NOFOLLOW on open(), and if

we get ELOOP then use readlink() to follow symlinks

• urselves in user space.
• This works surprisingly well, as we simply recurse into our

no-symlinks-allowed-outside the share open code with the new path from the link target.

• The same must be done for directory opens as well.

Opening Windows to a Wider World

Opening Windows to a Wider World

Welcome to my Nightmare

• Amazingly, the fix was written in about a week and modulo

a few bugs discovered later by Ralph Böhme worked almost immediately, and passed 99% of the tests.

• A few directory listing tests, and most of the shadow_copy2

VFS module tests failed - easy to fix I thought.

–10 minutes to fix the directory tests. –Mostly assumptions about current working directory being

the root of the share.

–Than I opened vfs_shadow_copy2.c. Oh..

Opening Windows to a Wider World

Opening Windows to a Wider World

• Shadow copy2 mutates incoming pathnames in

unimaginable ways, including use of regular expressions.

• It is designed to explicitly allow access to areas of the

filesystem outside of the share definition, if that is where the snapshot versions were stored.

–This module had been ignored since tridge left, as it mostly

worked and no one wanted to change it.

–Eventually I had to completely rewrite most of it.

Opening Windows to a Wider World

Opening Windows to a Wider World

Time Pressure

• Google Zero Day initiative gives a 90-day deadline, with
• ne 14-day extension allowed.

–Unfortunately, due to shadow_copy2 we needed the

extension.

• Prerequisite fixes were trickled into the code under the

pretext of fixing changed $cwd for “backup intent” code.

–Kept the actual “real” fix much cleaner and easier to

review.

• Uri Simchoni, Ralph Böhme, Andreas Schneider helped

with testing, additional fixes and back-ports to vendor supported versions. We couldn't have hit the deadline without all their work !

Opening Windows to a Wider World

Opening Windows to a Wider World

Coda

• After the fix went out, a vendor who shall remain nameless

but is well known to be a GPL-violator on Samba contacted me and complained the fix broke their proprietary VFS module.

–(What is the German word for schadenfreude ?).

• I am using this as a teachable event, as had their module

been upstream in the tree it would have been my problem to fix, not theirs :-).

• Ralph came up with some interesting ideas to prevent this

from happening again, using the chroot() system call to restrict access to an exported region of the filesystem.

Opening Windows to a Wider World

Opening Windows to a Wider World

Questions and Comments ? Email: jra@samba.org Slides available at: https://sambaxp.com