15-Minute Linux DFIR Triage Dr. Phil Polstra Bloomsburg University - - PowerPoint PPT Presentation

15-Minute Linux DFIR Triage

• Dr. Phil Polstra

Bloomsburg University of Pennsylvania

What is this talk about?

• Determining with some certainty if you have been hacked
• In a matter of minutes
• With minimal disturbance to the subject system
• Automating the process with shell scripting

Why should you care?

• Someone calls you about a suspected breach
• You need to need to figure out if they were hacked
• Quickly so as to avoid further harm to your client
• Without destroying evidence
• Without taking down a critical machine

Roadmap

• Opening a case
• Talking to the users
• Mounting known-good binaries
• Minimizing disturbance to the system
• Collecting Data
• Automation with scripting
• Next steps if there is a breach

Opening a Case

• Decide on case name (date?)
• Create a case folder on your laptop
• Start making entries in your notebook
• Bound notebook with numbered pages
• Easy to carry
• Hard to insert/remove pages
• No batteries required

First Talk to the Users

• They know the situation better than you
• Might be able to tell a false alarm before digging in
• Why did you call me?
• What they suspect
• No internal experts or policy to use outsider?
• Why do you think there was an incident?
• Everything they know about the subject system

USB Response Drive

• Contains known-good binaries
• Minimum /bin, /sbin, /lib for same architecture
• Might also grab /usr/sbin, /usr/bin, /usr/lib
• Must be on an ext2, ext3, or ext4 partition
• Could contain a bootable Linux on another partition
• This partition will probably be FAT
• Should be first partition
• See http://linuxforensicsbook.com/hdvideos.html Chapter 1

Mounting Known-Good Binaries

• Insert response drive
• Exec your bash binary
• Set path to your binaries (and only your binaries)
• Set LD_LIBRARY_PATH
• Run all shell scripts as bash <script>
• Don't use she-bang (#!) in scripts!

Demo: Mounting Binaries

Minimize Disturbance to System

• You will always change the system a little
• Goal is to
• Minimize memory footprint
• Never write to subject media
• Two basic options
• Save everything to your USB response drive
• Send it over the network

Sending data over the network

• Better than USB drive due to caching
• Use netcat
• Create a listener for “log” information on forensics workstation
• Send “log” information from client
• Also create a listener for interesting files on forensics

workstation

– Spawn a new listener when files are sent

Setting Up Log Listener

• netcat -k -l 9999 >> case-log.txt
• (-k) keep alive (-l) listen (>>) append
• From subject
• {command} | netcat {forensic ws IP} 9999
• Let's use shell scripting to automate this
• Shell not Python because we want to minimize memory

footprint

Automating the Log Listener

usage () { echo "usage: $0 <case number>" echo "Simple script to create case folder and start listeners" exit 1 } if [ $# -lt 1 ] ; then usage else echo "Starting case $1" fi #if the directory doesn't exist create it if [ ! -d $1 ] ; then mkdir $1 fi # create the log listener `nc -k -l 4444 >> $1/log.txt` & echo "Started log listener for case $1 on $(date)" | nc localhost 4444 # start the file listener `./start-file-listener.sh $1` &

Automating the Log Client-Part 1

usage () { echo "usage: $0 <IP> [log port] [fn port] [ft port]" exit 1 } # did you specify a file? if [ $# -lt 1 ] ; then usage fi export RHOST=$1 if [ $# -gt 1 ] ; then export RPORT=$2 else export RPORT=4444 fi

if [ $# -gt 2 ] ; then export RFPORT=$3 else export RFPORT=5555 fi if [ $# -gt 3 ] ; then export RFTPORT=$4 else export RFTPORT=5556 fi

Automating the Log Client – Part 2

# defaults primarily for testing [ -z "$RHOST" ] && { export RHOST=localhost; } [ -z "$RPORT" ] && { export RPORT=4444; } usage () { echo "usage: $0 <command or script>" echo "Simple script to send a log entry to listener" exit 1 } # did you specify a command? if [ $# -lt 1 ] ; then usage else echo -e "++++Sending log for $@ at $(date) ++++\n $($@) \n----end----\n" | nc $RHOST $RPORT fi

Automating Sending Files

• Listener on forensics workstation listens for file name
• When a new file name is received
• Create a new listener to receive file
• Redirect file to one with correct name
• Also log in the main case log (optional)
• On the client side
• File name is sent
• After brief pause send file to data listener port

Automating the File Listener

usage () { echo "usage: $0 <case name>" echo "Simple script to start a file listener" exit 1 } # did you specify a file? if [ $# -lt 1 ] ; then usage fi

while true do filename=$(nc -l 5555) nc -l 5556 > $1/$(basename $filename) done

Automating the File Client

# defaults primarily for testing [ -z "$RHOST" ] && { export RHOST=localhost; } [ -z "$RPORT" ] && { export RPORT=4444; } [ -z "$RFPORT" ] && { export RFPORT=5555; } [ -z "$RFTPORT" ] && { export RFTPORT=5556; } usage () { echo "usage: $0 <filename>" echo "Simple script to send a file to listener" exit 1 } # did you specify a file? if [ $# -lt 1 ] ; then usage fi #log it echo "Attempting to send file $1 at $(date)" | nc $RHOST $RPORT #send name echo $(basename $1) | nc $RHOST $RFPORT #give it time sleep 5 nc $RHOST $RFTPORT < $1

Cleaning Up

# close the case and clean up the listeners echo "Shutting down listeners at $(date) at user request" | nc localhost 4444 killall start-case.sh killall start-file-listener.sh killall nc

Collecting Data

• Date (date)
• Clock skew on subject
• Time zone on subject
• Kernel version (uname -a)
• Needed for memory analysis
• Might be useful for researching vulnerabilities

Collecting Data (continued)

• Network interfaces (ifconfig -a)
• Any new interfaces?
• Strange addresses assigned?
• Network connections (netstat -anp)
• Connects to suspicious Internet addresses?
• Strange localhost connections?
• Suspicious ports?
• Programs on wrong ports (i.e malware on port 80)

Collecting Data (continued)

• Open files (lsof -V)
• What programs are using certain files/ports
• Might fail if malware installed
• Running processes (ps -ef and/or ps -aux)
• Things run as root that shouldn't be
• No login accounts logged in and running things
• Might fail if malware installed

Collecting Data (continued)

• Routing info (netstat -rn and route)
• Routed through new interface
• New gateways
• Conflicting results = malware
• Failure to run = malware

Collecting Data (continued)

• Mounted filesystems (mount and df)
• Things mounted that shouldn't be (especially tempfs)
• Mount options that have changed
• Filesystem filling up
• Disagreement = malware
• Kernel modules (lsmod)
• New device drivers
• Modules that have changed

Collecting Data (continued)

• Who is logged in now (w)
• System accounts that shouldn't be allowed to login
• Who has been logging in (last)
• System accounts that shouldn't be allowed to login
• Accounts that don't normally use this machine
• Failed logins (lastb)
• Repeated failures then success = password cracked

Collecting Data (continued)

• User login info (send /etc/passwd and /etc/shadow)
• Newly created login
• System accounts with shells and home directories
• Accounts with ID 0
• Accounts with passwords that shouldn't be there

Putting It Together with a Script

usage () { echo "usage: $0 [listening host]" echo "Simple script to send a log entry to listener" exit 1 } # did you specify a listener IP? if [ $# -gt 1 ] || [ "$1" == "--help" ] ; then usage fi # did you specify a listener IP? if [ "$1" != "" ] ; then source setup-client.sh $1 fi

# now collect some info! send-log.sh date send-log.sh uname -a send-log.sh ifconfig -a send-log.sh netstat -anp send-log.sh lsof -V send-log.sh ps -ef send-log.sh netstat -rn send-log.sh route send-log.sh lsmod send-log.sh df send-log.sh mount send-log.sh w send-log.sh last send-log.sh lastb send-log.sh cat /etc/passwd send-log.sh cat /etc/shadow

Running the Initial Scan

Have I been hacked?

Who is Johnn?

/etc/passwd

Why do these accounts have passwords?

/etc/shadow

Who's been logging in?

Results from last

Who failed to login?

Results from lastb

Looks Like They Were Hacked Now What?

Live Analysis

• Use techniques described to
• Grab file metadata for key directories (/sbin, /bin, etc.)
• Grab users' command history
• Get system log files
• Get hashes of suspicious files
• Dump RAM
• Must compile LiME (off subject machine!)
• Risky – can cause a crash

Dead Analysis

• Unless the machine absolutely cannot be taken offline it is

strongly recommended that you shut it down and get a filesystem image

• If you cannot shutdown the machine
• You can still get a filesystem image with dcfldd
• You probably cannot use this evidence in court

More on Dead Analysis

• Filesystem analysis is much more mature and powerful

than memory analysis

• The Linux support in Volatility is somewhat lacking
• Relatively new addition to a new tool
• Seems to fall down a lot with late 3.x and 4.x kernels
• None of the investigators I've talked to could come up with

a case where evidence existed only in memory

Finding Out More

• Heard there was a new book out (1 kg+ of knowledge)
• Resources on http://linuxforensicsbook.com
• Harass me on Twitter @ppolstra

Questions?