Ben Stienstra

• Capture all internet traffic to a 3GB RAM disk, sync to NAS.
• Sync dumps to NAS for archiving and later analysis.
• Connected to UPS to prevent data loss due to power outage.
• Monitor the dump process, restart if needed, send alert via email.

                     +------------+
                     | NAS        |
                     |            |
                     +---------^--+
                               |   
                               |  rsync over NFS
                               |   
                     +---------+--+
Mirror port traff.   | APU1C4     |
          +---------->            |
                     +------------+

• APU1C4 (4GB mem)
• 16GB SSD

• Upgrade BIOS via PXE or USB.

• Reserve a hostname / IP address in DNS.

• CentOS 6.5 on APU - KickStart file
• Check IP address order to configure.

  ip a

• Log in with SSH, switch to root.
• Set hostname. Edit /etc/sysconfig/network

  HOSTNAME=localhost.localdomain

• Configure fixed IP address. Edit /etc/sysconfig/network/ifcfg-eth0.
• Enable second interface without ip IP address /etc/sysconfig/network/ifcfg-eth1

  ONBOOT=yes
  NM_CONTROLLED=no
  BOOTPROTO=static

• CentOS 6.5 SSD configuration
• Install and configure NTPd.

  yum intall ntp
  
  #edit servers in /etc/ntp.conf
  
  ntpdate ntp1.polaire.nl
  chkconfig ntpd on 
  service ntpd start

• Set SELinux booleans. (this, again, took me several hours to find out SELinux was the problem…)

  setsebool -P rsync_use_nfs 1
  setsebool -P rsync_export_all_ro 1

• Reboot, test if everything is ok.

• Add tmpfs filesystem to /etc/fstab.

  tmpfs /mnt/ram tmpfs size=3g 0 0

• Create mount point.

  mkdir /mnt/ram

• Mount filesystem.

  mount /mnt/ram

• Reserve a few TB's on NAS
• Install NFS tools.

  yum install nfs-utils
  chkconfig nfs on
  chkconfig rpcbind on
  service rpcbind start
  service nfs start

• Create mount point.

  mkdir /mnt/pcap

• Add mount to /etc/fstab

  nfsserver:/volume1/pcap   /mnt/pcap   nfs	intr	0 0

• Install rsync.

  yum install rsync

• Install monit from source, monit in epel repo is old…

  cd /root
  yum install pam-devel openssl-devel
  
  wget http://mmonit.com/monit/dist/monit-5.8.1.tar.gz
  tar zxvf monit-5.8.1.tar.gz
  
  cd monit-5.8.1
  ./configure
  make
  make install
  cp system/startup/rc.monit /etc/init.d/monit
  edit : MONIT=/usr/local/bin/monit
  set logfile /var/log/monit.log
  
  cp monitrc /usr/local/etc
  edit and add: include /usr/local/etc/monit.d/*
  mkdir -p /usr/local/etc/monit.d
  
  
  chmod +x /etc/init.d/monit
  chkconfig --add monit
  

• Create monit config file /usr/local/etc/monit.d/tcpdump

  check process tcpdump matching "tcpdump"
     start program = "/bin/nice -n -10 /usr/sbin/tcpdump -i eth1 -s0 -nn -G60 -w /mnt/ram/tcpdump-eth1-%Y-%m-%d_%H:%M:%S.pcap"
     stop program = "/usr/bin/pkill tcpdump"
     if 5 restarts within 5 cycles then timeout

• If you want alerts via mail, add this to /etc/monit.conf

  set mailserver localhost
  set alert your@mail.com

• Start monit.

  service monit start

• Kill tcpdump, to test if monit will restart tcpdump and send an alert.

  pkill tcpdump

• CentOS 6.5 on APU - post install

• create shutdown script /etc/init.d/rsync-capture

  #!/bin/bash
  #
  # chkconfig: - 95 05
  
  ### BEGIN INIT INFO
  # Provides: rsync-capture
  # Required-Stop: $network $local_fs $remote_fs
  # Required-Start: $syslog
  # Default-Start: 3
  # Default-Stop: 0 1 6
  # Short-Description: sync RAM disk to NAS
  # Description: rsync network captures from RAM disk to NAS
  ### END INIT INFO
  
  # Source function library.
  . /etc/init.d/functions
  
  # Source networking configuration.
  . /etc/sysconfig/network
  
  # Lock file
  prog=rsync-capture
  lockfile=/var/lock/subsys/$prog
  ramdisk=/mnt/ram
  nas=/mnt/pcap
  
  stop() {
          [ "$EUID" != "0" ] && exit 4
          echo -n $"$prog, syncing RAM disk to NAS before shutdown."
          echo " ---- STOP runlevel: `/sbin/runlevel` date:  `date`" >> /var/log/rsync.log
          /usr/bin/rsync --quiet -a --log-file=/var/log/rsync.log $ramdisk/ $nas
          RETVAL=$?
          echo
          [ $RETVAL -eq 0 ] && rm -f $lockfile
          return $RETVAL
  }
  
  start() {
          [ "$EUID" != "0" ] && exit 4
          echo -n $"$prog, sync not needed at start-up."
          echo " ---- START runlevel: `/sbin/runlevel` date: `date`" >> /var/log/rsync.log
          RETVAL=$?
          echo
          [ $RETVAL -eq 0 ] && touch $lockfile
          return $RETVAL
  }
  
  sync() {
          [ "$EUID" != "0" ] && exit 4
          # Sync all files but last (the one tcpdump is writing to) to nas dir and year/day subdirs.
          # remove source files from RAM disk, if sync was succesful.
  
          # first create directory structure
          mkdir -p $nas/`date +%Y/%m/%d`
        
          # sync files
          ls $ramdisk | sort -t. -k2 | head -n -1 | /usr/bin/rsync --quiet -a --remove-source-files --log-file=/var/log/rsync.log --files-from=- $ramdisk/ $nas/`date +%Y/%m/%d`
          RETVAL=$?
          return $RETVAL
  }
  
  # See how we were called.
  case "$1" in
    stop) 
          stop
          ;;
    start)
          start
          ;;
    sync) 
          sync
          ;;
  
    *)
          echo $"Usage: $0 {start|stop|sync}"
          exit 2
  esac
  
  
  

• Run a cronjob to sync data to NAS every minute. Add to /etc/crontab

  * *  *  *  * root service rsync-capture sync