parsep-extend-range.pl Your friendly neighborhood PCAP parser

Part of my job is to watch our organizations IDS/IPS sensors and respond to alerts. If you know who I am, I don't think that it will come as a surprise to you that we use snort to monitor our little corner of the metaverse.

A problem I was continuously running into when investigating alarms generated with snort as with almost any IDS/IPS is that often times all you have to work with is payload from the single packet that triggered the alarm and maybe logs on a server. I realize that if you configure a snort rule to do so, you can tag other interesting traffic from a would-be attacker after an alarm has fired. Sguil provides a means for extracting an entire session based on the packet that generated the alarm via log_packets.sh. You can also use sguil to extract individual sessions not generated by an alert if you are logging with sancp.

While this is fine it lacked some features that I really desired such as the output generated by honeysnap , afterglow, argus , etc. The other issue that I ran into is that even though I was performing full packet capture, tools such as ethereal and tcpdump don't support wildcard 's so if you have directory with 100 1 gig pcaps and you want to extract all traffic to/from an attacker out of these pcaps you are shit out of luck unless you pass your bpf to tcpdump one pcap at a time. The first iteration of parsep was to search every pcap that matched a given a wild card for traffic from a specified ip address /netmask, save the data to a temp file and then put it all back together into one uber-pcap using mergecap for analysis. This was fine if you only have to parse a couple of gigs of pcaps, but obviously does not scale well to a couple hundred gigs of pcaps. The latest iteration that I'm releasing optionally uses session data from argus based on time to determine in what pcaps traffic from our attacker resides. So instead of having to parse 200 gigs of pcaps you may only have to parse 1 gig of session data and based on it's output parse only the 10 1 gig pcaps that actually contain data from the attacker. Hmmm that is a really horrible description, how about we just get to some examples.

Usage: parsep-extend-range.pl <ip addy 2 find> <netmask> argus <last x number of files 2 search or 0 for all matching argusfilemask> <path to argus file> pcap <last x number of files 2 search or 0 for all matching pcapfilemask> <pcapfilemask>

Example: perl parsep-extend-range.pl 127.0.0.1 32 argus 0 /var/log/sessiondata/argusfile.* pcap 0 /var/log/fullcap/daemonlogger.*

./parsep-extend-range.pl 127.0.0.1 32 argus 2 /var/log/sessiondata/argusoutput.* pcap 0 /var/log/fullcap/daemonlogger.*

removed /var/log/sessiondata/argusoutput.ra.5 from argus search list
removed /var/log/sessiondata/argusoutput.ra.4 from argus search list
removed /var/log/sessiondata/argusoutput.ra.3 from argus search list
removed /var/log/sessiondata/argusoutput.ra.2 from argus search list
removed 4 argus files out of 6 so that only last 2 remain
argusfile here is /var/log/sessiondata/argusoutput.ra.1 /var/log/sessiondata/argusoutput.ra
making dir to store data
not removing any files from pcap filemask array
finding proper pcaps from argus session data in file /var/log/sessiondata/argusoutput.ra.1
finding proper pcaps from argus session data in file /var/log/sessiondata/argusoutput.ra
file list before dup removal
putting file /var/log/fullcap/daemonlogger.pcap.1190640570 into new file array
putting file /var/log/fullcap/daemonlogger.pcap.1190640570 into new file array
putting file /var/log/fullcap/daemonlogger.pcap.1190640570 into new file array
putting file /var/log/fullcap/daemonlogger.pcap.1190640570 into new file array
putting file /var/log/fullcap/daemonlogger.pcap.1190647914 into new file array
putting file /var/log/fullcap/daemonlogger.pcap.1190647914 into new file array
putting file /var/log/fullcap/daemonlogger.pcap.1190647914 into new file array
putting file /var/log/fullcap/daemonlogger.pcap.1190647914 into new file array
putting file /var/log/fullcap/daemonlogger.pcap.1190647914 into new file array
putting file /var/log/fullcap/daemonlogger.pcap.1190648607 into new file array
revised file list
/var/log/fullcap/daemonlogger.pcap.1190640570
/var/log/fullcap/daemonlogger.pcap.1190647914
/var/log/fullcap/daemonlogger.pcap.1190648607
searching for 127.0.0.1/32 in file /var/log/fullcap/daemonlogger.pcap.1190640570
reading from file /var/log/fullcap/daemonlogger.pcap.1190640570, link-type EN10MB (Ethernet)
searching for 127.0.0.1/32 in file /var/log/fullcap/daemonlogger.pcap.1190647914
reading from file /var/log/fullcap/daemonlogger.pcap.1190647914, link-type EN10MB (Ethernet)
searching for 127.0.0.1/32 in file /var/log/fullcap/daemonlogger.pcap.1190648607
reading from file /var/log/fullcap/daemonlogger.pcap.1190648607, link-type EN10MB (Ethernet)
merging pcaps
generating connection graph using afterglow
reading from file 127.0.0.11190681713/127.0.0.1.pcap, link-type EN10MB (Ethernet)
No property file specified, using default settings.
Not a color:
generating argus file from merged pcap
outputing session data to text file
exporting tcpflowdata
generating honeysnap data
creating tarball 127.0.0.11190681713.tgz
removing temp data
we got you now sucka

In the example above we are storing about five day's of session data with argus and 200 gigs of pcaps using daemonlogger. I know that it is dumb but the script requires that full pcaps have a filename of something.something.date for example daemonlogger.pcap.1190647914 The script I use for daemonlogger is included below

#!/bin/sh
. /etc/init.d/functions
case "$1" in
start)
echo -n "Starting daemonlogger: "
/usr/local/bin/daemonlogger -r -d -i br0 -l /var/log/fullcap/ -m 200 -S 1515
touch /var/lock/daemonlogger
sleep 3
echo
;;
stop)
echo -n "Stopping daemonlogger: "
killproc daemonlogger
rm -f /var/lock/daemonlogger
echo
;;
restart)
$0 stop
$0 start
;;
status)
status daemonlogger
;;
*)
echo "Usage: $0 {start|stop|restart|status}"
exit 1
esac

exit 0

The script I use for argus is below along with the logrotate file. I use this because argus doesn't have any sort of built-in ringbuffer functionality like daemonlogger or tshark. Because the logrotate file causes our older session data to be added first to the array, we flip the array to cause new session data to be moved to the front of the array.

#!/bin/sh
. /etc/init.d/functions
case "$1" in
start)
echo -n "Starting argus: "
/usr/local/sbin/argus -d -p -c -J -w /var/log/sessiondata/argusoutput.ra -i br0
touch /var/lock/argus
sleep 3
echo
;;
stop)
echo -n "Stopping argus: "
kill `cat /var/run/argus.pid`
echo
;;
restart)
$0 stop
$0 start
;;
status)
status argus
;;
*)
echo "Usage: $0 {start|stop|restart|status}"
exit 1
esac

exit 0

Logrotate file

/var/log/sessiondata/argusoutput.ra
{
rotate 5
missingok
nocompress
daily
postrotate
/etc/init.d/argusd restart >/dev/null 2&>1
endscript
}

I hope somebody finds this useful ;-), If not, oh well it is useful to me. If you look at the script you can see how easy it is to add support for almost any tool that you want to run the merged pcap through. If you don't understand the importance of collecting full content captures along with session data I suggest that you pickup one or all of Richard Bejtlich's books on NSM.

Matt Jonkman of bleedingthreats.net was kind enough to host the script for me. He is truly a king among men ;-)....

download
parsep-extend-range.pl.bz2