15-Minute Linux DFIR Triage Dr. Phil Polstra Bloomsburg University - PowerPoint PPT Presentation

15-Minute Linux DFIR Triage Dr. Phil Polstra Bloomsburg University of Pennsylvania

What is this talk about? ● Determining with some certainty if you have been hacked ● In a matter of minutes ● With minimal disturbance to the subject system ● Automating the process with shell scripting

Why should you care? ● Someone calls you about a suspected breach ● You need to need to figure out if they were hacked ● Quickly so as to avoid further harm to your client ● Without destroying evidence ● Without taking down a critical machine

Roadmap ● Opening a case ● Talking to the users ● Mounting known-good binaries ● Minimizing disturbance to the system ● Collecting Data ● Automation with scripting ● Next steps if there is a breach

Opening a Case ● Decide on case name (date?) ● Create a case folder on your laptop ● Start making entries in your notebook ● Bound notebook with numbered pages ● Easy to carry ● Hard to insert/remove pages ● No batteries required

First Talk to the Users ● They know the situation better than you ● Might be able to tell a false alarm before digging in ● Why did you call me? ● What they suspect ● No internal experts or policy to use outsider? ● Why do you think there was an incident? ● Everything they know about the subject system

USB Response Drive ● Contains known-good binaries ● Minimum /bin, /sbin, /lib for same architecture ● Might also grab /usr/sbin, /usr/bin, /usr/lib ● Must be on an ext2, ext3, or ext4 partition ● Could contain a bootable Linux on another partition ● This partition will probably be FAT ● Should be first partition ● See http://linuxforensicsbook.com/hdvideos.html Chapter 1

Mounting Known-Good Binaries ● Insert response drive ● Exec your bash binary ● Set path to your binaries (and only your binaries) ● Set LD_LIBRARY_PATH ● Run all shell scripts as bash <script> ● Don't use she-bang (#!) in scripts!

Demo: Mounting Binaries

Minimize Disturbance to System ● You will always change the system a little ● Goal is to ● Minimize memory footprint ● Never write to subject media ● Two basic options ● Save everything to your USB response drive ● Send it over the network

Sending data over the network ● Better than USB drive due to caching ● Use netcat ● Create a listener for “log” information on forensics workstation ● Send “log” information from client ● Also create a listener for interesting files on forensics workstation – Spawn a new listener when files are sent

Setting Up Log Listener ● netcat -k -l 9999 >> case-log.txt ● (-k) keep alive (-l) listen (>>) append ● From subject ● {command} | netcat {forensic ws IP} 9999 ● Let's use shell scripting to automate this ● Shell not Python because we want to minimize memory footprint

Automating the Log Listener usage () { #if the directory doesn't exist create it echo "usage: $0 <case number>" if [ ! -d $1 ] ; then echo "Simple script to create case folder and mkdir $1 start listeners" fi exit 1 } # create the log listener `nc -k -l 4444 >> $1/log.txt` & if [ $# -lt 1 ] ; then echo "Started log listener for case $1 on $(date)" usage | nc localhost 4444 else echo "Starting case $1" # start the file listener fi `./start-file-listener.sh $1` &

Automating the Log Client-Part 1 usage () { if [ $# -gt 2 ] ; then echo "usage: $0 <IP> [log port] [fn port] [ft port]" export RFPORT=$3 exit 1 else } export RFPORT=5555 # did you specify a file? if [ $# -lt 1 ] ; then fi usage if [ $# -gt 3 ] ; then fi export RFTPORT=$4 export RHOST=$1 if [ $# -gt 1 ] ; then else export RPORT=$2 export RFTPORT=5556 else fi export RPORT=4444 fi

Automating the Log Client – Part 2 # defaults primarily for testing [ -z "$RHOST" ] && { export RHOST=localhost; } [ -z "$RPORT" ] && { export RPORT=4444; } usage () { echo "usage: $0 <command or script>" echo "Simple script to send a log entry to listener" exit 1 } # did you specify a command? if [ $# -lt 1 ] ; then usage else echo -e "++++Sending log for $@ at $(date) ++++\n $($@) \n----end----\n" | nc $RHOST $RPORT fi

Automating Sending Files ● Listener on forensics workstation listens for file name ● When a new file name is received ● Create a new listener to receive file ● Redirect file to one with correct name ● Also log in the main case log (optional) ● On the client side ● File name is sent ● After brief pause send file to data listener port

Automating the File Listener usage () { while true echo "usage: $0 <case name>" do echo "Simple script to start a file listener" filename=$(nc -l 5555) exit 1 nc -l 5556 > $1/$(basename } $filename) done # did you specify a file? if [ $# -lt 1 ] ; then usage fi

Automating the File Client # defaults primarily for testing # did you specify a file? if [ $# -lt 1 ] ; then [ -z "$RHOST" ] && { export RHOST=localhost; } usage [ -z "$RPORT" ] && { export RPORT=4444; } fi [ -z "$RFPORT" ] && { export RFPORT=5555; } [ -z "$RFTPORT" ] && { export RFTPORT=5556; } #log it echo "Attempting to send file $1 at $(date)" | nc usage () { $RHOST $RPORT echo "usage: $0 <filename>" #send name echo "Simple script to send a file to listener" echo $(basename $1) | nc $RHOST $RFPORT exit 1 #give it time } sleep 5 nc $RHOST $RFTPORT < $1

Cleaning Up # close the case and clean up the listeners echo "Shutting down listeners at $(date) at user request" | nc localhost 4444 killall start-case.sh killall start-file-listener.sh killall nc

Collecting Data ● Date (date) ● Clock skew on subject ● Time zone on subject ● Kernel version (uname -a) ● Needed for memory analysis ● Might be useful for researching vulnerabilities

Collecting Data (continued) ● Network interfaces (ifconfig -a) ● Any new interfaces? ● Strange addresses assigned? ● Network connections (netstat -anp) ● Connects to suspicious Internet addresses? ● Strange localhost connections? ● Suspicious ports? ● Programs on wrong ports (i.e malware on port 80)

Collecting Data (continued) ● Open files (lsof -V) ● What programs are using certain files/ports ● Might fail if malware installed ● Running processes (ps -ef and/or ps -aux) ● Things run as root that shouldn't be ● No login accounts logged in and running things ● Might fail if malware installed

Collecting Data (continued) ● Routing info (netstat -rn and route) ● Routed through new interface ● New gateways ● Conflicting results = malware ● Failure to run = malware

Collecting Data (continued) ● Mounted filesystems (mount and df) ● Things mounted that shouldn't be (especially tempfs) ● Mount options that have changed ● Filesystem filling up ● Disagreement = malware ● Kernel modules (lsmod) ● New device drivers ● Modules that have changed

Collecting Data (continued) ● Who is logged in now (w) ● System accounts that shouldn't be allowed to login ● Who has been logging in (last) ● System accounts that shouldn't be allowed to login ● Accounts that don't normally use this machine ● Failed logins (lastb) ● Repeated failures then success = password cracked

Collecting Data (continued) ● User login info (send /etc/passwd and /etc/shadow) ● Newly created login ● System accounts with shells and home directories ● Accounts with ID 0 ● Accounts with passwords that shouldn't be there

Putting It Together with a Script usage () { # now collect some info! send-log.sh date echo "usage: $0 [listening host]" send-log.sh uname -a echo "Simple script to send a log entry to listener" send-log.sh ifconfig -a exit 1 send-log.sh netstat -anp } send-log.sh lsof -V send-log.sh ps -ef # did you specify a listener IP? send-log.sh netstat -rn if [ $# -gt 1 ] || [ "$1" == "--help" ] ; then send-log.sh route usage send-log.sh lsmod fi send-log.sh df send-log.sh mount # did you specify a listener IP? send-log.sh w if [ "$1" != "" ] ; then send-log.sh last send-log.sh lastb source setup-client.sh $1 send-log.sh cat /etc/passwd fi send-log.sh cat /etc/shadow

Running the Initial Scan

Have I been hacked?

Who is Johnn? /etc/passwd

Why do these accounts have passwords? /etc/shadow

Who's been logging in? Results from last

Who failed to login? Results from lastb

Looks Like They Were Hacked Now What?

Live Analysis ● Use techniques described to ● Grab file metadata for key directories (/sbin, /bin, etc.) ● Grab users' command history ● Get system log files ● Get hashes of suspicious files ● Dump RAM ● Must compile LiME (off subject machine!) ● Risky – can cause a crash

Dead Analysis ● Unless the machine absolutely cannot be taken offline it is strongly recommended that you shut it down and get a filesystem image ● If you cannot shutdown the machine ● You can still get a filesystem image with dcfldd ● You probably cannot use this evidence in court