Ben Stienstra

Linux, Unix, network, radio and more...

User Tools

Site Tools


full_packet_capture_system

~~TOC~~

Full packet capture system

Concept

  • Capture all internet traffic to a 3GB RAM disk, sync to NAS.
  • Sync dumps to NAS for archiving and later analysis.
  • Connected to UPS to prevent data loss due to power outage.
  • Monitor the dump process, restart if needed, send alert via email.
                     +------------+
                     | NAS        |
                     |            |
                     +---------^--+
                               |   
                               |  rsync over NFS
                               |   
                     +---------+--+
Mirror port traff.   | APU1C4     |
          +---------->            |
                     +------------+

Hardware

  • APU1C4 (4GB mem)
  • 16GB SSD

Upgrade BIOS

  • Upgrade BIOS via PXE or USB.

Network

  • Reserve a hostname / IP address in DNS.
interface description
eth0 management
eth1 mirrored traffic (rx/tx)

Install and configure OS

  • Check IP address order to configure.
    ip a
  • Log in with SSH, switch to root.
  • Set hostname. Edit /etc/sysconfig/network
    HOSTNAME=localhost.localdomain
  • Configure fixed IP address. Edit /etc/sysconfig/network/ifcfg-eth0.
  • Enable second interface without ip IP address /etc/sysconfig/network/ifcfg-eth1
    ONBOOT=yes
    NM_CONTROLLED=no
    BOOTPROTO=static
  • Install and configure NTPd.
    yum intall ntp
    
    #edit servers in /etc/ntp.conf
    
    ntpdate ntp1.polaire.nl
    chkconfig ntpd on 
    service ntpd start
  • Set SELinux booleans. (this, again, took me several hours to find out SELinux was the problem…)
    setsebool -P rsync_use_nfs 1
    setsebool -P rsync_export_all_ro 1
  • Reboot, test if everything is ok.

Configure full packet capture system

Create 3GB RAM disk

  • Add tmpfs filesystem to /etc/fstab.
    tmpfs /mnt/ram tmpfs size=3g 0 0
  • Create mount point.
    mkdir /mnt/ram
  • Mount filesystem.
    mount /mnt/ram

Create NFS mount to NAS

  • Reserve a few TB's on NAS
  • Install NFS tools.
    yum install nfs-utils
    chkconfig nfs on
    chkconfig rpcbind on
    service rpcbind start
    service nfs start
  • Create mount point.
    mkdir /mnt/pcap
  • Add mount to /etc/fstab
    nfsserver:/volume1/pcap   /mnt/pcap   nfs	intr	0 0

rsync

  • Install rsync.
    yum install rsync

monit

  • Install monit from source, monit in epel repo is old…
    cd /root
    yum install pam-devel openssl-devel
    
    wget http://mmonit.com/monit/dist/monit-5.8.1.tar.gz
    tar zxvf monit-5.8.1.tar.gz
    
    cd monit-5.8.1
    ./configure
    make
    make install
    cp system/startup/rc.monit /etc/init.d/monit
    edit : MONIT=/usr/local/bin/monit
    set logfile /var/log/monit.log
    
    cp monitrc /usr/local/etc
    edit and add: include /usr/local/etc/monit.d/*
    mkdir -p /usr/local/etc/monit.d
    
    
    chmod +x /etc/init.d/monit
    chkconfig --add monit
    
  • Create monit config file /usr/local/etc/monit.d/tcpdump
    check process tcpdump matching "tcpdump"
       start program = "/bin/nice -n -10 /usr/sbin/tcpdump -i eth1 -s0 -nn -G60 -w /mnt/ram/tcpdump-eth1-%Y-%m-%d_%H:%M:%S.pcap"
       stop program = "/usr/bin/pkill tcpdump"
       if 5 restarts within 5 cycles then timeout
  • If you want alerts via mail, add this to /etc/monit.conf
    set mailserver localhost
    set alert your@mail.com
  • Start monit.
    service monit start
  • Kill tcpdump, to test if monit will restart tcpdump and send an alert.
    pkill tcpdump

Configure mail

rsync init script

  • create shutdown script /etc/init.d/rsync-capture
    #!/bin/bash
    #
    # chkconfig: - 95 05
    
    ### BEGIN INIT INFO
    # Provides: rsync-capture
    # Required-Stop: $network $local_fs $remote_fs
    # Required-Start: $syslog
    # Default-Start: 3
    # Default-Stop: 0 1 6
    # Short-Description: sync RAM disk to NAS
    # Description: rsync network captures from RAM disk to NAS
    ### END INIT INFO
    
    # Source function library.
    . /etc/init.d/functions
    
    # Source networking configuration.
    . /etc/sysconfig/network
    
    # Lock file
    prog=rsync-capture
    lockfile=/var/lock/subsys/$prog
    ramdisk=/mnt/ram
    nas=/mnt/pcap
    
    stop() {
            [ "$EUID" != "0" ] && exit 4
            echo -n $"$prog, syncing RAM disk to NAS before shutdown."
            echo " ---- STOP runlevel: `/sbin/runlevel` date:  `date`" >> /var/log/rsync.log
            /usr/bin/rsync --quiet -a --log-file=/var/log/rsync.log $ramdisk/ $nas
            RETVAL=$?
            echo
            [ $RETVAL -eq 0 ] && rm -f $lockfile
            return $RETVAL
    }
    
    start() {
            [ "$EUID" != "0" ] && exit 4
            echo -n $"$prog, sync not needed at start-up."
            echo " ---- START runlevel: `/sbin/runlevel` date: `date`" >> /var/log/rsync.log
            RETVAL=$?
            echo
            [ $RETVAL -eq 0 ] && touch $lockfile
            return $RETVAL
    }
    
    sync() {
            [ "$EUID" != "0" ] && exit 4
            # Sync all files but last (the one tcpdump is writing to) to nas dir and year/day subdirs.
            # remove source files from RAM disk, if sync was succesful.
    
            # first create directory structure
            mkdir -p $nas/`date +%Y/%m/%d`
          
            # sync files
            ls $ramdisk | sort -t. -k2 | head -n -1 | /usr/bin/rsync --quiet -a --remove-source-files --log-file=/var/log/rsync.log --files-from=- $ramdisk/ $nas/`date +%Y/%m/%d`
            RETVAL=$?
            return $RETVAL
    }
    
    # See how we were called.
    case "$1" in
      stop) 
            stop
            ;;
      start)
            start
            ;;
      sync) 
            sync
            ;;
    
      *)
            echo $"Usage: $0 {start|stop|sync}"
            exit 2
    esac
    
    
    

Enable rsync

  • Run a cronjob to sync data to NAS every minute. Add to /etc/crontab
    * *  *  *  * root service rsync-capture sync
full_packet_capture_system.txt · Last modified: 2014/06/17 10:48 by admin