full_packet_capture

Full packet capture system

Full packet capture system

Concept

Capture all internet traffic to a 3GB RAM disk, sync to NAS.
Sync dumps to NAS for archiving and later analysis.
Connected to UPS to prevent data loss due to power outage.
Monitor the dump process, restart if needed, send alert via email.

                     +------------+
                     | NAS        |
                     |            |
                     +---------^--+
                               |   
                               |  rsync over NFS
                               |   
                     +---------+--+
Mirror port traff.   | APU1C4     |
          +---------->            |
                     +------------+

Hardware

APU1C4 (4GB mem)
16GB SSD

Upgrade BIOS

Upgrade BIOS via PXE or USB.

Network

Reserve a hostname / IP address in DNS.

interface	description
eth0	management
eth1	mirrored traffic (rx/tx)

Install and configure OS

CentOS 6.5 on APU - KickStart file
Check IP address order to configure.
```
ip a
```
Log in with SSH, switch to root.
Set hostname. Edit /etc/sysconfig/network
```
HOSTNAME=localhost.localdomain
```
Configure fixed IP address. Edit /etc/sysconfig/network/ifcfg-eth0.
Enable second interface without ip IP address /etc/sysconfig/network/ifcfg-eth1
```
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=static
```
CentOS 6.5 SSD configuration

Install and configure NTPd.

yum intall ntp

#edit servers in /etc/ntp.conf

ntpdate ntp1.polaire.nl
chkconfig ntpd on 
service ntpd start

Set SELinux booleans. (this, again, took me several hours to find out SELinux was the problem…)
```
setsebool -P rsync_use_nfs 1
setsebool -P rsync_export_all_ro 1
```
Reboot, test if everything is ok.

Configure full packet capture system

Create 3GB RAM disk

Add tmpfs filesystem to /etc/fstab.
```
tmpfs /mnt/ram tmpfs size=3g 0 0
```
Create mount point.
```
mkdir /mnt/ram
```
Mount filesystem.
```
mount /mnt/ram
```

Create NFS mount to NAS

Reserve a few TB's on NAS

Install NFS tools.

yum install nfs-utils
chkconfig nfs on
chkconfig rpcbind on
service rpcbind start
service nfs start

Create mount point.
```
mkdir /mnt/pcap
```

Add mount to /etc/fstab

nfsserver:/volume1/pcap   /mnt/pcap   nfs	intr	0 0

rsync

Install rsync.
```
yum install rsync
```

monit

Install monit from source, monit in epel repo is old…

cd /root
yum install pam-devel openssl-devel

wget http://mmonit.com/monit/dist/monit-5.8.1.tar.gz
tar zxvf monit-5.8.1.tar.gz

cd monit-5.8.1
./configure
make
make install
cp system/startup/rc.monit /etc/init.d/monit
edit : MONIT=/usr/local/bin/monit
set logfile /var/log/monit.log

cp monitrc /usr/local/etc
edit and add: include /usr/local/etc/monit.d/*
mkdir -p /usr/local/etc/monit.d


chmod +x /etc/init.d/monit
chkconfig --add monit

Create monit config file /usr/local/etc/monit.d/tcpdump

check process tcpdump matching "tcpdump"
   start program = "/bin/nice -n -10 /usr/sbin/tcpdump -i eth1 -s0 -nn -G60 -w /mnt/ram/tcpdump-eth1-%Y-%m-%d_%H:%M:%S.pcap"
   stop program = "/usr/bin/pkill tcpdump"
   if 5 restarts within 5 cycles then timeout

If you want alerts via mail, add this to /etc/monit.conf
```
set mailserver localhost
set alert your@mail.com
```
Start monit.
```
service monit start
```
Kill tcpdump, to test if monit will restart tcpdump and send an alert.
```
pkill tcpdump
```

Configure mail

CentOS 6.5 on APU - post install

rsync init script

create shutdown script /etc/init.d/rsync-capture

#!/bin/bash
#
# chkconfig: - 95 05

### BEGIN INIT INFO
# Provides: rsync-capture
# Required-Stop: $network $local_fs $remote_fs
# Required-Start: $syslog
# Default-Start: 3
# Default-Stop: 0 1 6
# Short-Description: sync RAM disk to NAS
# Description: rsync network captures from RAM disk to NAS
### END INIT INFO

# Source function library.
. /etc/init.d/functions

# Source networking configuration.
. /etc/sysconfig/network

# Lock file
prog=rsync-capture
lockfile=/var/lock/subsys/$prog
ramdisk=/mnt/ram
nas=/mnt/pcap

stop() {
        [ "$EUID" != "0" ] && exit 4
        echo -n $"$prog, syncing RAM disk to NAS before shutdown."
        echo " ---- STOP runlevel: `/sbin/runlevel` date:  `date`" >> /var/log/rsync.log
        /usr/bin/rsync --quiet -a --log-file=/var/log/rsync.log $ramdisk/ $nas
        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && rm -f $lockfile
        return $RETVAL
}

start() {
        [ "$EUID" != "0" ] && exit 4
        echo -n $"$prog, sync not needed at start-up."
        echo " ---- START runlevel: `/sbin/runlevel` date: `date`" >> /var/log/rsync.log
        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && touch $lockfile
        return $RETVAL
}

sync() {
        [ "$EUID" != "0" ] && exit 4
        # Sync all files but last (the one tcpdump is writing to) to nas dir and year/day subdirs.
        # remove source files from RAM disk, if sync was succesful.

        # first create directory structure
        mkdir -p $nas/`date +%Y/%m/%d`
      
        # sync files
        ls $ramdisk | sort -t. -k2 | head -n -1 | /usr/bin/rsync --quiet -a --remove-source-files --log-file=/var/log/rsync.log --files-from=- $ramdisk/ $nas/`date +%Y/%m/%d`
        RETVAL=$?
        return $RETVAL
}

# See how we were called.
case "$1" in
  stop) 
        stop
        ;;
  start)
        start
        ;;
  sync) 
        sync
        ;;

  *)
        echo $"Usage: $0 {start|stop|sync}"
        exit 2
esac

Enable rsync

Run a cronjob to sync data to NAS every minute. Add to /etc/crontab
```
* *  *  *  * root service rsync-capture sync
```

Table of Contents